2023-03-24 21:53:33,237 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430 2023-03-24 21:53:33,250 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-24 21:53:33,266 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-24 21:53:33,266 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de, deleteOnExit=true 2023-03-24 21:53:33,266 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-24 21:53:33,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/test.cache.data in system properties and HBase conf 2023-03-24 21:53:33,267 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/hadoop.tmp.dir in system properties and HBase conf 2023-03-24 21:53:33,268 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/hadoop.log.dir in system properties and HBase conf 2023-03-24 21:53:33,268 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-24 21:53:33,269 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-24 21:53:33,269 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-24 21:53:33,389 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-24 21:53:33,820 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-24 21:53:33,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-24 21:53:33,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-24 21:53:33,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-24 21:53:33,825 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 21:53:33,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-24 21:53:33,826 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-24 21:53:33,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-24 21:53:33,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 21:53:33,827 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-24 21:53:33,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/nfs.dump.dir in system properties and HBase conf 2023-03-24 21:53:33,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/java.io.tmpdir in system properties and HBase conf 2023-03-24 21:53:33,828 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-24 21:53:33,829 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-24 21:53:33,829 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-24 21:53:34,443 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 21:53:34,447 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 21:53:34,754 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-24 21:53:34,937 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-24 21:53:34,954 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 21:53:35,001 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 21:53:35,034 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/java.io.tmpdir/Jetty_localhost_44063_hdfs____ey5e6v/webapp 2023-03-24 21:53:35,181 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:44063 2023-03-24 21:53:35,217 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-24 21:53:35,218 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-24 21:53:35,704 WARN [Listener at localhost/45841] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 21:53:35,842 WARN [Listener at localhost/45841] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-24 21:53:35,865 WARN [Listener at localhost/45841] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-24 21:53:35,873 INFO [Listener at localhost/45841] log.Slf4jLog(67): jetty-6.1.26 2023-03-24 21:53:35,879 INFO [Listener at localhost/45841] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/java.io.tmpdir/Jetty_localhost_36181_datanode____zco8pa/webapp 2023-03-24 21:53:36,001 INFO [Listener at localhost/45841] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:36181 2023-03-24 21:53:36,375 WARN [Listener at localhost/40745] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-24 21:53:36,912 DEBUG [Listener at localhost/40745] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430 2023-03-24 21:53:36,932 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc729aa5b1f4b521c: Processing first storage report for DS-0eccf154-8714-4e8c-b2f3-81fdfc718733 from datanode 6db8e130-b292-4c29-b205-f31c95509b69 2023-03-24 21:53:36,936 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc729aa5b1f4b521c: from storage DS-0eccf154-8714-4e8c-b2f3-81fdfc718733 node DatanodeRegistration(127.0.0.1:36995, datanodeUuid=6db8e130-b292-4c29-b205-f31c95509b69, infoPort=38677, infoSecurePort=0, ipcPort=40745, storageInfo=lv=-57;cid=testClusterID;nsid=531009107;c=1679694814530), blocks: 0, hasStaleStorage: true, processing time: 4 msecs, invalidatedBlocks: 0 2023-03-24 21:53:36,936 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0xc729aa5b1f4b521c: Processing first storage report for DS-9d30d607-1c7d-43b4-9e96-10ae31b66449 from datanode 6db8e130-b292-4c29-b205-f31c95509b69 2023-03-24 21:53:36,936 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0xc729aa5b1f4b521c: from storage DS-9d30d607-1c7d-43b4-9e96-10ae31b66449 node DatanodeRegistration(127.0.0.1:36995, datanodeUuid=6db8e130-b292-4c29-b205-f31c95509b69, infoPort=38677, infoSecurePort=0, ipcPort=40745, storageInfo=lv=-57;cid=testClusterID;nsid=531009107;c=1679694814530), blocks: 0, hasStaleStorage: false, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-24 21:53:37,001 INFO [Listener at localhost/40745] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de/zookeeper_0, clientPort=57724, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-24 21:53:37,017 INFO [Listener at localhost/40745] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=57724 2023-03-24 21:53:37,027 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:37,029 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:37,693 INFO [Listener at localhost/40745] util.FSUtils(471): Created version file at hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420 with version=8 2023-03-24 21:53:37,694 INFO [Listener at localhost/40745] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/hbase-staging 2023-03-24 21:53:38,022 INFO [Listener at localhost/40745] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-24 21:53:38,510 INFO [Listener at localhost/40745] client.ConnectionUtils(127): master/jenkins-hbase9:0 server-side Connection retries=45 2023-03-24 21:53:38,542 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:38,542 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:38,542 INFO [Listener at localhost/40745] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 21:53:38,543 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:38,543 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 21:53:38,684 INFO [Listener at localhost/40745] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 21:53:38,765 DEBUG [Listener at localhost/40745] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-24 21:53:38,858 INFO [Listener at localhost/40745] ipc.NettyRpcServer(120): Bind to /172.31.2.10:40479 2023-03-24 21:53:38,868 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:38,870 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:38,890 INFO [Listener at localhost/40745] zookeeper.RecoverableZooKeeper(93): Process identifier=master:40479 connecting to ZooKeeper ensemble=127.0.0.1:57724 2023-03-24 21:53:38,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:404790x0, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 21:53:38,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:40479-0x1011885c6c20000 connected 2023-03-24 21:53:38,958 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 21:53:38,959 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 21:53:38,962 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 21:53:38,970 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=40479 2023-03-24 21:53:38,970 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=40479 2023-03-24 21:53:38,971 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=40479 2023-03-24 21:53:38,972 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=40479 2023-03-24 21:53:38,973 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=40479 2023-03-24 21:53:38,979 INFO [Listener at localhost/40745] master.HMaster(444): hbase.rootdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420, hbase.cluster.distributed=false 2023-03-24 21:53:39,053 INFO [Listener at localhost/40745] client.ConnectionUtils(127): regionserver/jenkins-hbase9:0 server-side Connection retries=45 2023-03-24 21:53:39,053 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:39,053 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:39,054 INFO [Listener at localhost/40745] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-24 21:53:39,054 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-24 21:53:39,054 INFO [Listener at localhost/40745] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-24 21:53:39,059 INFO [Listener at localhost/40745] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-24 21:53:39,062 INFO [Listener at localhost/40745] ipc.NettyRpcServer(120): Bind to /172.31.2.10:45673 2023-03-24 21:53:39,064 INFO [Listener at localhost/40745] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-24 21:53:39,070 DEBUG [Listener at localhost/40745] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-24 21:53:39,072 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:39,075 INFO [Listener at localhost/40745] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:39,076 INFO [Listener at localhost/40745] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:45673 connecting to ZooKeeper ensemble=127.0.0.1:57724 2023-03-24 21:53:39,080 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:456730x0, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-24 21:53:39,081 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:45673-0x1011885c6c20001 connected 2023-03-24 21:53:39,081 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 21:53:39,082 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 21:53:39,083 DEBUG [Listener at localhost/40745] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-24 21:53:39,086 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=45673 2023-03-24 21:53:39,087 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=45673 2023-03-24 21:53:39,087 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=45673 2023-03-24 21:53:39,088 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=45673 2023-03-24 21:53:39,091 DEBUG [Listener at localhost/40745] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=45673 2023-03-24 21:53:39,093 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:39,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 21:53:39,103 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:39,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 21:53:39,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:39,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-24 21:53:39,123 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 21:53:39,124 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase9.apache.org,40479,1679694817836 from backup master directory 2023-03-24 21:53:39,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-24 21:53:39,128 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:39,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-24 21:53:39,129 WARN [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 21:53:39,129 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:39,132 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-24 21:53:39,134 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-24 21:53:39,625 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/hbase.id with ID: 463eba31-7d82-483d-b437-8f4a4bbddd77 2023-03-24 21:53:39,673 INFO [master/jenkins-hbase9:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-24 21:53:39,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:39,731 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x78fa30d3 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:39,767 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3ddf8a67, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:39,793 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:53:39,795 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-24 21:53:39,816 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-24 21:53:39,816 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-24 21:53:39,818 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:39,823 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:39,824 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 21:53:39,856 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store-tmp 2023-03-24 21:53:40,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:40,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 21:53:40,287 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:53:40,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:53:40,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 21:53:40,287 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:53:40,287 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:53:40,288 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 21:53:40,289 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/WALs/jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:40,310 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C40479%2C1679694817836, suffix=, logDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/WALs/jenkins-hbase9.apache.org,40479,1679694817836, archiveDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/oldWALs, maxLogs=10 2023-03-24 21:53:40,361 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK] 2023-03-24 21:53:40,369 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:40,429 INFO [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/WALs/jenkins-hbase9.apache.org,40479,1679694817836/jenkins-hbase9.apache.org%2C40479%2C1679694817836.1679694820320 2023-03-24 21:53:40,429 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK]] 2023-03-24 21:53:40,430 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:53:40,430 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:40,433 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,435 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,489 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,495 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-24 21:53:40,518 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-24 21:53:40,530 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:40,535 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,537 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,551 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-24 21:53:40,556 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:53:40,557 INFO [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63284578, jitterRate=-0.05698630213737488}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 21:53:40,557 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 21:53:40,559 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-24 21:53:40,581 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-24 21:53:40,581 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-24 21:53:40,584 INFO [master/jenkins-hbase9:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-24 21:53:40,586 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-24 21:53:40,621 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 34 msec 2023-03-24 21:53:40,621 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-24 21:53:40,648 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-24 21:53:40,653 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-24 21:53:40,677 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-24 21:53:40,680 INFO [master/jenkins-hbase9:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-24 21:53:40,682 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-24 21:53:40,687 INFO [master/jenkins-hbase9:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-24 21:53:40,691 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-24 21:53:40,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:40,695 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-24 21:53:40,695 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-24 21:53:40,711 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-24 21:53:40,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 21:53:40,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-24 21:53:40,717 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:40,717 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase9.apache.org,40479,1679694817836, sessionid=0x1011885c6c20000, setting cluster-up flag (Was=false) 2023-03-24 21:53:40,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:40,738 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-24 21:53:40,739 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:40,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:40,751 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-24 21:53:40,752 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:40,755 WARN [master/jenkins-hbase9:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.hbase-snapshot/.tmp 2023-03-24 21:53:40,796 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(952): ClusterId : 463eba31-7d82-483d-b437-8f4a4bbddd77 2023-03-24 21:53:40,800 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-24 21:53:40,807 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-24 21:53:40,807 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-24 21:53:40,814 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-24 21:53:40,815 DEBUG [RS:0;jenkins-hbase9:45673] zookeeper.ReadOnlyZKClient(139): Connect 0x1217d297 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:40,826 DEBUG [RS:0;jenkins-hbase9:45673] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1cd02ea5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:40,826 DEBUG [RS:0;jenkins-hbase9:45673] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@18efb9e7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-03-24 21:53:40,850 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase9:45673 2023-03-24 21:53:40,853 INFO [RS:0;jenkins-hbase9:45673] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-24 21:53:40,853 INFO [RS:0;jenkins-hbase9:45673] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-24 21:53:40,853 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1023): About to register with Master. 2023-03-24 21:53:40,856 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase9.apache.org,40479,1679694817836 with isa=jenkins-hbase9.apache.org/172.31.2.10:45673, startcode=1679694819052 2023-03-24 21:53:40,872 DEBUG [RS:0;jenkins-hbase9:45673] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-24 21:53:40,878 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=5, maxPoolSize=5 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase9:0, corePoolSize=10, maxPoolSize=10 2023-03-24 21:53:40,888 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:40,889 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-03-24 21:53:40,889 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:40,890 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679694850890 2023-03-24 21:53:40,893 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-24 21:53:40,897 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 21:53:40,897 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-24 21:53:40,903 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-24 21:53:40,904 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 21:53:40,912 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-24 21:53:40,912 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-24 21:53:40,912 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-24 21:53:40,912 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-24 21:53:40,915 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:40,916 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-24 21:53:40,918 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-24 21:53:40,919 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-24 21:53:40,921 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-24 21:53:40,922 INFO [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-24 21:53:40,923 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1679694820923,5,FailOnTimeoutGroup] 2023-03-24 21:53:40,924 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1679694820924,5,FailOnTimeoutGroup] 2023-03-24 21:53:40,924 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:40,924 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-24 21:53:40,925 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:40,925 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:40,930 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:59863, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-24 21:53:40,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.ServerManager(394): Registering regionserver=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:40,956 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420 2023-03-24 21:53:40,956 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:45841 2023-03-24 21:53:40,956 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-24 21:53:40,961 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 21:53:40,962 DEBUG [RS:0;jenkins-hbase9:45673] zookeeper.ZKUtil(162): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:40,962 WARN [RS:0;jenkins-hbase9:45673] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-24 21:53:40,962 INFO [RS:0;jenkins-hbase9:45673] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 21:53:40,963 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1947): logDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/WALs/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:40,963 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase9.apache.org,45673,1679694819052] 2023-03-24 21:53:40,971 DEBUG [RS:0;jenkins-hbase9:45673] zookeeper.ZKUtil(162): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:40,981 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-24 21:53:40,989 INFO [RS:0;jenkins-hbase9:45673] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-24 21:53:41,012 INFO [RS:0;jenkins-hbase9:45673] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-24 21:53:41,016 INFO [RS:0;jenkins-hbase9:45673] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-24 21:53:41,017 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,018 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-24 21:53:41,025 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,025 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,025 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,025 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase9:0, corePoolSize=2, maxPoolSize=2 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,026 DEBUG [RS:0;jenkins-hbase9:45673] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase9:0, corePoolSize=1, maxPoolSize=1 2023-03-24 21:53:41,027 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,027 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,027 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,043 INFO [RS:0;jenkins-hbase9:45673] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-24 21:53:41,045 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,45673,1679694819052-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:41,061 INFO [RS:0;jenkins-hbase9:45673] regionserver.Replication(203): jenkins-hbase9.apache.org,45673,1679694819052 started 2023-03-24 21:53:41,061 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1638): Serving as jenkins-hbase9.apache.org,45673,1679694819052, RpcServer on jenkins-hbase9.apache.org/172.31.2.10:45673, sessionid=0x1011885c6c20001 2023-03-24 21:53:41,062 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-24 21:53:41,062 DEBUG [RS:0;jenkins-hbase9:45673] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:41,062 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,45673,1679694819052' 2023-03-24 21:53:41,062 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:41,063 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:41,063 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-24 21:53:41,063 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-24 21:53:41,063 DEBUG [RS:0;jenkins-hbase9:45673] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:41,063 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase9.apache.org,45673,1679694819052' 2023-03-24 21:53:41,064 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-24 21:53:41,064 DEBUG [RS:0;jenkins-hbase9:45673] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-24 21:53:41,064 DEBUG [RS:0;jenkins-hbase9:45673] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-24 21:53:41,065 INFO [RS:0;jenkins-hbase9:45673] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-24 21:53:41,065 INFO [RS:0;jenkins-hbase9:45673] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-24 21:53:41,176 INFO [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45673%2C1679694819052, suffix=, logDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/WALs/jenkins-hbase9.apache.org,45673,1679694819052, archiveDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/oldWALs, maxLogs=32 2023-03-24 21:53:41,196 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK] 2023-03-24 21:53:41,202 INFO [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/WALs/jenkins-hbase9.apache.org,45673,1679694819052/jenkins-hbase9.apache.org%2C45673%2C1679694819052.1679694821178 2023-03-24 21:53:41,202 DEBUG [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK]] 2023-03-24 21:53:41,337 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 21:53:41,339 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-24 21:53:41,339 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420 2023-03-24 21:53:41,353 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:41,356 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 21:53:41,358 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/info 2023-03-24 21:53:41,359 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 21:53:41,360 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,361 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 21:53:41,363 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/rep_barrier 2023-03-24 21:53:41,364 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 21:53:41,365 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,365 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 21:53:41,367 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/table 2023-03-24 21:53:41,368 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 21:53:41,369 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,371 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740 2023-03-24 21:53:41,372 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740 2023-03-24 21:53:41,376 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:53:41,378 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 21:53:41,381 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:53:41,382 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63788442, jitterRate=-0.04947814345359802}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:53:41,383 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 21:53:41,383 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 21:53:41,383 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 21:53:41,383 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 21:53:41,383 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 21:53:41,383 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 21:53:41,384 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 21:53:41,385 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 21:53:41,390 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-24 21:53:41,390 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-24 21:53:41,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-24 21:53:41,411 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-24 21:53:41,413 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-24 21:53:41,565 DEBUG [jenkins-hbase9:40479] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-24 21:53:41,568 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,45673,1679694819052, state=OPENING 2023-03-24 21:53:41,577 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-24 21:53:41,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:41,579 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 21:53:41,581 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:53:41,767 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:41,769 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-24 21:53:41,773 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38230, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-24 21:53:41,785 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-24 21:53:41,785 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-24 21:53:41,789 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase9.apache.org%2C45673%2C1679694819052.meta, suffix=.meta, logDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/WALs/jenkins-hbase9.apache.org,45673,1679694819052, archiveDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/oldWALs, maxLogs=32 2023-03-24 21:53:41,808 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK] 2023-03-24 21:53:41,819 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/WALs/jenkins-hbase9.apache.org,45673,1679694819052/jenkins-hbase9.apache.org%2C45673%2C1679694819052.meta.1679694821791.meta 2023-03-24 21:53:41,819 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36995,DS-0eccf154-8714-4e8c-b2f3-81fdfc718733,DISK]] 2023-03-24 21:53:41,820 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:53:41,822 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-24 21:53:41,846 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-24 21:53:41,853 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-24 21:53:41,859 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-24 21:53:41,859 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-24 21:53:41,860 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-24 21:53:41,863 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-24 21:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/info 2023-03-24 21:53:41,865 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/info 2023-03-24 21:53:41,866 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-24 21:53:41,867 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,867 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-24 21:53:41,869 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/rep_barrier 2023-03-24 21:53:41,869 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/rep_barrier 2023-03-24 21:53:41,869 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-24 21:53:41,870 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,871 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-24 21:53:41,872 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/table 2023-03-24 21:53:41,872 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/table 2023-03-24 21:53:41,873 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-24 21:53:41,874 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:41,876 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740 2023-03-24 21:53:41,878 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740 2023-03-24 21:53:41,882 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:53:41,885 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-24 21:53:41,886 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66318605, jitterRate=-0.011775776743888855}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:53:41,887 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-24 21:53:41,896 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679694821758 2023-03-24 21:53:41,913 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-24 21:53:41,913 INFO [RS_OPEN_META-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-24 21:53:41,914 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase9.apache.org,45673,1679694819052, state=OPEN 2023-03-24 21:53:41,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-24 21:53:41,918 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-24 21:53:41,922 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-24 21:53:41,922 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase9.apache.org,45673,1679694819052 in 337 msec 2023-03-24 21:53:41,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-24 21:53:41,927 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 525 msec 2023-03-24 21:53:41,932 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.1260 sec 2023-03-24 21:53:41,932 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679694821932, completionTime=-1 2023-03-24 21:53:41,932 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-24 21:53:41,933 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-24 21:53:41,996 DEBUG [hconnection-0x770a4cd5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:42,000 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38240, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:42,021 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-24 21:53:42,021 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679694882021 2023-03-24 21:53:42,021 INFO [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679694942021 2023-03-24 21:53:42,022 INFO [master/jenkins-hbase9:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 88 msec 2023-03-24 21:53:42,046 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40479,1679694817836-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:42,046 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40479,1679694817836-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:42,046 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40479,1679694817836-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:42,048 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase9:40479, period=300000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:42,049 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-24 21:53:42,055 DEBUG [master/jenkins-hbase9:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-24 21:53:42,064 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-24 21:53:42,066 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-24 21:53:42,077 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-24 21:53:42,080 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:53:42,083 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:53:42,105 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,107 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a empty. 2023-03-24 21:53:42,108 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,108 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-24 21:53:42,129 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-24 21:53:42,132 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 40b7a2bac1d0cd056af86198ad6d9f0a, NAME => 'hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:53:42,544 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:42,545 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 40b7a2bac1d0cd056af86198ad6d9f0a, disabling compactions & flushes 2023-03-24 21:53:42,545 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,545 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,545 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. after waiting 0 ms 2023-03-24 21:53:42,545 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,545 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,545 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 40b7a2bac1d0cd056af86198ad6d9f0a: 2023-03-24 21:53:42,550 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:53:42,565 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679694822553"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694822553"}]},"ts":"1679694822553"} 2023-03-24 21:53:42,592 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:53:42,594 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:53:42,598 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694822594"}]},"ts":"1679694822594"} 2023-03-24 21:53:42,602 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-24 21:53:42,613 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=40b7a2bac1d0cd056af86198ad6d9f0a, ASSIGN}] 2023-03-24 21:53:42,616 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=40b7a2bac1d0cd056af86198ad6d9f0a, ASSIGN 2023-03-24 21:53:42,617 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=40b7a2bac1d0cd056af86198ad6d9f0a, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:53:42,768 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=40b7a2bac1d0cd056af86198ad6d9f0a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:42,769 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679694822768"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694822768"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694822768"}]},"ts":"1679694822768"} 2023-03-24 21:53:42,773 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 40b7a2bac1d0cd056af86198ad6d9f0a, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:53:42,933 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,933 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 40b7a2bac1d0cd056af86198ad6d9f0a, NAME => 'hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:53:42,934 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,935 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:42,935 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,935 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,937 INFO [StoreOpener-40b7a2bac1d0cd056af86198ad6d9f0a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,939 DEBUG [StoreOpener-40b7a2bac1d0cd056af86198ad6d9f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/info 2023-03-24 21:53:42,939 DEBUG [StoreOpener-40b7a2bac1d0cd056af86198ad6d9f0a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/info 2023-03-24 21:53:42,939 INFO [StoreOpener-40b7a2bac1d0cd056af86198ad6d9f0a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 40b7a2bac1d0cd056af86198ad6d9f0a columnFamilyName info 2023-03-24 21:53:42,940 INFO [StoreOpener-40b7a2bac1d0cd056af86198ad6d9f0a-1] regionserver.HStore(310): Store=40b7a2bac1d0cd056af86198ad6d9f0a/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:42,942 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,942 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,947 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:53:42,950 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:53:42,951 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 40b7a2bac1d0cd056af86198ad6d9f0a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69579423, jitterRate=0.03681419789791107}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-24 21:53:42,951 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 40b7a2bac1d0cd056af86198ad6d9f0a: 2023-03-24 21:53:42,953 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a., pid=6, masterSystemTime=1679694822926 2023-03-24 21:53:42,957 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,957 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:53:42,958 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=40b7a2bac1d0cd056af86198ad6d9f0a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:42,959 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679694822958"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694822958"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694822958"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694822958"}]},"ts":"1679694822958"} 2023-03-24 21:53:42,965 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-24 21:53:42,965 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 40b7a2bac1d0cd056af86198ad6d9f0a, server=jenkins-hbase9.apache.org,45673,1679694819052 in 189 msec 2023-03-24 21:53:42,969 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-24 21:53:42,969 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=40b7a2bac1d0cd056af86198ad6d9f0a, ASSIGN in 352 msec 2023-03-24 21:53:42,970 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:53:42,971 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694822970"}]},"ts":"1679694822970"} 2023-03-24 21:53:42,973 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-24 21:53:42,977 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:53:42,979 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 910 msec 2023-03-24 21:53:42,980 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-24 21:53:42,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-24 21:53:42,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:53:43,017 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-24 21:53:43,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 21:53:43,039 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 29 msec 2023-03-24 21:53:43,051 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-24 21:53:43,063 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-24 21:53:43,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-03-24 21:53:43,078 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-24 21:53:43,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-24 21:53:43,082 INFO [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.953sec 2023-03-24 21:53:43,084 INFO [master/jenkins-hbase9:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-24 21:53:43,085 INFO [master/jenkins-hbase9:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-24 21:53:43,085 INFO [master/jenkins-hbase9:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-24 21:53:43,086 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40479,1679694817836-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-24 21:53:43,087 INFO [master/jenkins-hbase9:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase9.apache.org,40479,1679694817836-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-24 21:53:43,097 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-24 21:53:43,103 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x0904a3c5 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,107 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23b8d36a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,120 DEBUG [hconnection-0x669432c5-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,131 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38256, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:43,141 INFO [Listener at localhost/40745] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:53:43,161 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=291, ProcessCount=168, AvailableMemoryMB=8966 2023-03-24 21:53:43,177 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:53:43,180 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44554, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:53:43,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:53:43,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:53:43,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:53:43,199 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:53:43,201 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:53:43,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-24 21:53:43,205 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,206 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 empty. 2023-03-24 21:53:43,208 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,208 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:53:43,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 21:53:43,230 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:53:43,232 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => a01d0abc8722ae28834e0262613a5783, NAME => 'TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing a01d0abc8722ae28834e0262613a5783, disabling compactions & flushes 2023-03-24 21:53:43,244 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. after waiting 0 ms 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,244 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,244 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:43,249 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:53:43,252 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694823252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694823252"}]},"ts":"1679694823252"} 2023-03-24 21:53:43,255 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:53:43,257 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:53:43,257 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694823257"}]},"ts":"1679694823257"} 2023-03-24 21:53:43,260 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:53:43,265 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, ASSIGN}] 2023-03-24 21:53:43,267 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, ASSIGN 2023-03-24 21:53:43,269 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:53:43,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 21:53:43,421 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a01d0abc8722ae28834e0262613a5783, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:43,421 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694823421"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694823421"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694823421"}]},"ts":"1679694823421"} 2023-03-24 21:53:43,425 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:53:43,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 21:53:43,584 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => a01d0abc8722ae28834e0262613a5783, NAME => 'TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:53:43,584 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:53:43,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,585 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,587 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,589 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A 2023-03-24 21:53:43,589 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A 2023-03-24 21:53:43,591 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:53:43,591 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a01d0abc8722ae28834e0262613a5783 columnFamilyName A 2023-03-24 21:53:43,592 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(310): Store=a01d0abc8722ae28834e0262613a5783/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:43,592 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,594 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B 2023-03-24 21:53:43,594 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B 2023-03-24 21:53:43,595 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:53:43,595 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a01d0abc8722ae28834e0262613a5783 columnFamilyName B 2023-03-24 21:53:43,596 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(310): Store=a01d0abc8722ae28834e0262613a5783/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:43,596 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,598 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C 2023-03-24 21:53:43,598 DEBUG [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C 2023-03-24 21:53:43,598 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:53:43,598 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region a01d0abc8722ae28834e0262613a5783 columnFamilyName C 2023-03-24 21:53:43,599 INFO [StoreOpener-a01d0abc8722ae28834e0262613a5783-1] regionserver.HStore(310): Store=a01d0abc8722ae28834e0262613a5783/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:53:43,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,603 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,606 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:53:43,608 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:43,612 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:53:43,612 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened a01d0abc8722ae28834e0262613a5783; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60147555, jitterRate=-0.10373158752918243}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:53:43,613 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:43,614 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., pid=11, masterSystemTime=1679694823578 2023-03-24 21:53:43,616 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,616 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:43,617 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=a01d0abc8722ae28834e0262613a5783, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:43,617 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694823617"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694823617"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694823617"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694823617"}]},"ts":"1679694823617"} 2023-03-24 21:53:43,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-24 21:53:43,623 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 in 196 msec 2023-03-24 21:53:43,626 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-24 21:53:43,627 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, ASSIGN in 358 msec 2023-03-24 21:53:43,628 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:53:43,628 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694823628"}]},"ts":"1679694823628"} 2023-03-24 21:53:43,630 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:53:43,633 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:53:43,635 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 438 msec 2023-03-24 21:53:43,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-24 21:53:43,825 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-24 21:53:43,830 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x4507c0c2 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,835 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7cbf3e48, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,837 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,840 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:43,843 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:53:43,846 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44566, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:53:43,854 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x09417ce8 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,858 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e2586ae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,859 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x258416be to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,864 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15826a66, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,864 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x634d1248 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,869 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4008a0b3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,869 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x6ec02d2b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,873 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@609c84d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,874 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x30a6dfbc to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,878 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@701d57b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,878 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5cc86371 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,883 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@16d69540, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,883 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5eb3f68d to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,911 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@41ffc5c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,912 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x2b52d1b4 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,916 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14bcd8d2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,916 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x3625ffd0 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:53:43,920 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bc747d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:53:43,958 DEBUG [hconnection-0x4896b578-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,959 DEBUG [hconnection-0x2b90131a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,964 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,972 DEBUG [hconnection-0x1330534b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,972 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38276, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:43,979 DEBUG [hconnection-0x139a00b7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,981 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38288, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:43,979 DEBUG [hconnection-0x79e1e112-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,988 DEBUG [hconnection-0x27e010-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,994 DEBUG [hconnection-0x1a5dd2ff-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:43,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:43,999 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,001 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,005 DEBUG [hconnection-0x43a25080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:53:44,006 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38308, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,016 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38316, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,023 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38324, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,023 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38322, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:44,034 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:44,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:44,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:44,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:44,035 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:38328, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:53:44,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:44,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,037 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:44,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:44,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:44,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:44,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:44,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:44,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:44,042 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:44,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:44,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:44,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:44,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:44,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:44,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:44,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:44,055 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:44,057 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 21:53:44,070 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:44,071 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:44,071 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:44,072 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:44,072 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:44,072 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:44,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:44,098 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:44,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694824154, max timestamp=1679694824197Num uniques 859; 2023-03-24 21:53:44,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:44,251 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824154, max timestamp=1679694824197Num uniques 859; 2023-03-24 21:53:44,252 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 21:53:44,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824154, max timestamp=1679694824190Num uniques 859; 2023-03-24 21:53:44,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 21:53:44,257 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/6d4fd6b7dd9b484fb901709b0f5186d6 2023-03-24 21:53:44,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694824292, max timestamp=1679694824310Num uniques 1009; 2023-03-24 21:53:44,325 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-24 21:53:44,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824292, max timestamp=1679694824310Num uniques 1018; 2023-03-24 21:53:44,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-24 21:53:44,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824292, max timestamp=1679694824316Num uniques 1009; 2023-03-24 21:53:44,333 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 21:53:44,375 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/3c867a3e2a1645158a3e1e7780007ba4 2023-03-24 21:53:44,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694824398, max timestamp=1679694824421Num uniques 1027; 2023-03-24 21:53:44,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 21:53:44,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694824398, max timestamp=1679694824421Num uniques 1009; 2023-03-24 21:53:44,441 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-24 21:53:44,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824398, max timestamp=1679694824421Num uniques 1009; 2023-03-24 21:53:44,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 21:53:44,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824463, max timestamp=1679694824521Num uniques 1036; 2023-03-24 21:53:44,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824463, max timestamp=1679694824521Num uniques 1009; 2023-03-24 21:53:44,534 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 21:53:44,533 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-24 21:53:44,536 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824463, max timestamp=1679694824521Num uniques 1036; 2023-03-24 21:53:44,537 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=6870, suffix heap size=135712,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=150 2023-03-24 21:53:44,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694824556, max timestamp=1679694824579Num uniques 1009; 2023-03-24 21:53:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-24 21:53:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694824556, max timestamp=1679694824579Num uniques 1009; 2023-03-24 21:53:44,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-24 21:53:44,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694824556, max timestamp=1679694824579Num uniques 1009; 2023-03-24 21:53:44,587 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=11220, suffix heap size=132904,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=245 2023-03-24 21:53:44,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:44,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694884624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:44,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694884625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,635 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694824613, max timestamp=1679694824626Num uniques 1104; 2023-03-24 21:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694824613, max timestamp=1679694824626Num uniques 1104; 2023-03-24 21:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694824613, max timestamp=1679694824626Num uniques 1104; 2023-03-24 21:53:44,636 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:53:44,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694824665, max timestamp=1679694824679Num uniques 1013; 2023-03-24 21:53:44,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 21:53:44,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694824665, max timestamp=1679694824679Num uniques 1013; 2023-03-24 21:53:44,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 21:53:44,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824672, max timestamp=1679694824679Num uniques 1013; 2023-03-24 21:53:44,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=6870, suffix heap size=133320,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=150 2023-03-24 21:53:44,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679694824694, max timestamp=1679694824731Num uniques 1072; 2023-03-24 21:53:44,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 21:53:44,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679694824694, max timestamp=1679694824731Num uniques 1072; 2023-03-24 21:53:44,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 21:53:44,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679694824694, max timestamp=1679694824731Num uniques 1009; 2023-03-24 21:53:44,740 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=7872, suffix heap size=132904,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=172 2023-03-24 21:53:44,789 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10566, totalHeapSize=24264, min timestamp=1679694824763, max timestamp=1679694824786Num uniques 1031; 2023-03-24 21:53:44,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=47206, new segment data size=10566, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-24 21:53:44,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10571, totalHeapSize=24264, min timestamp=1679694824763, max timestamp=1679694824786Num uniques 1031; 2023-03-24 21:53:44,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47211, new segment data size=10571, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-24 21:53:44,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694824763, max timestamp=1679694824786Num uniques 1031; 2023-03-24 21:53:44,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-24 21:53:44,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:44,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679694824806, max timestamp=1679694824816Num uniques 1090; 2023-03-24 21:53:44,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1679694824806, max timestamp=1679694824816Num uniques 1090; 2023-03-24 21:53:44,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=10980, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-24 21:53:44,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:44,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-24 21:53:44,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694884813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824806, max timestamp=1679694824816Num uniques 1090; 2023-03-24 21:53:44,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49910, new segment data size=6870, suffix heap size=141328,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=150 2023-03-24 21:53:44,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694884815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,839 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=8 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/4ae9a01f7e3342c0a1597b877d86cc55 2023-03-24 21:53:44,852 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/6d4fd6b7dd9b484fb901709b0f5186d6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6 2023-03-24 21:53:44,866 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6, entries=100, sequenceid=8, filesize=9.3 K 2023-03-24 21:53:44,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679694824852, max timestamp=1679694824864Num uniques 1099; 2023-03-24 21:53:44,871 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824852, max timestamp=1679694824864Num uniques 1009; 2023-03-24 21:53:44,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694824852, max timestamp=1679694824864Num uniques 1099; 2023-03-24 21:53:44,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=6870, suffix heap size=142264,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=150 2023-03-24 21:53:44,873 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46224, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-03-24 21:53:44,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-24 21:53:44,875 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/3c867a3e2a1645158a3e1e7780007ba4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4 2023-03-24 21:53:44,900 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4, entries=100, sequenceid=8, filesize=9.3 K 2023-03-24 21:53:44,904 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/4ae9a01f7e3342c0a1597b877d86cc55 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55 2023-03-24 21:53:44,918 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55, entries=100, sequenceid=8, filesize=9.3 K 2023-03-24 21:53:44,922 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=38.11 KB/39022 for a01d0abc8722ae28834e0262613a5783 in 864ms, sequenceid=8, compaction requested=false 2023-03-24 21:53:44,924 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 21:53:44,926 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:44,926 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:44,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:44,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:44,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:44,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:44,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:44,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:44,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:44,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:44,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:44,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:44,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d2c58ec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:44,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:44,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:44,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:44,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:44,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:44,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:44,950 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,950 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:44,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:44,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:44,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:44,963 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:44,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:44,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:44,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:44,963 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:44,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=51.53 KB heapSize=137.30 KB 2023-03-24 21:53:44,965 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:44,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:44,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:44,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:44,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:44,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:44,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:44,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:44,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:44,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:44,975 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:45,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:45,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:45,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,022 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,023 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:45,023 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:45,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679694824991, max timestamp=1679694825043Num uniques 859; 2023-03-24 21:53:45,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-24 21:53:45,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679694824991, max timestamp=1679694825043Num uniques 859; 2023-03-24 21:53:45,050 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-24 21:53:45,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=559, cellSize=25595, totalHeapSize=58376, min timestamp=1679694824991, max timestamp=1679694825043Num uniques 859; 2023-03-24 21:53:45,051 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=25595, suffix heap size=117064,new segment heap size=58376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=559 2023-03-24 21:53:45,065 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/5546db00b6c340458880157e38a520b5 2023-03-24 21:53:45,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694885090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694885093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,114 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694885094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694885096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694885096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,143 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:45,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:45,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.70 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/75aa24a4edd84cfb9cb597c9f6229edd 2023-03-24 21:53:45,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:45,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:45,195 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:45,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,199 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:45,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:45,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,202 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:45,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:45,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:45,205 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694885203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694885204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:45,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:45,219 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:45,219 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:45,220 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,220 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:45,221 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:45,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:45,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:45,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:45,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:45,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694885221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694885221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694885221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:45,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:45,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c479d59[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:45,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:45,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:45,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:45,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:45,245 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,245 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:45,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:45,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:45,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:45,256 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:45,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:45,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:45,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:45,256 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:45,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.70 KB at sequenceid=204 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/18282588fc734896b707e2e9eae6b6d7 2023-03-24 21:53:45,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,268 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:45,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,280 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:45,280 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:45,279 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,280 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,280 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,281 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/5546db00b6c340458880157e38a520b5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5 2023-03-24 21:53:45,292 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5, entries=350, sequenceid=204, filesize=21.1 K 2023-03-24 21:53:45,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/75aa24a4edd84cfb9cb597c9f6229edd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd 2023-03-24 21:53:45,307 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd, entries=150, sequenceid=204, filesize=11.8 K 2023-03-24 21:53:45,317 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/18282588fc734896b707e2e9eae6b6d7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7 2023-03-24 21:53:45,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7, entries=150, sequenceid=204, filesize=11.8 K 2023-03-24 21:53:45,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~51.53 KB/52762, heapSize ~137.25 KB/140544, currentSize=154.31 KB/158010 for a01d0abc8722ae28834e0262613a5783 in 366ms, sequenceid=204, compaction requested=false 2023-03-24 21:53:45,332 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:45,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:45,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:45,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:45,391 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:45,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:45,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:45,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,396 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:45,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:45,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:45,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:45,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,400 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:45,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:45,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:45,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:45,404 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=154.31 KB heapSize=407.25 KB 2023-03-24 21:53:45,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:45,405 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:45,405 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:45,405 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:45,405 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:45,405 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:45,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:45,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:45,471 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/cbc563f1bf4f49f8b14963c14d3f4940 2023-03-24 21:53:45,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694885441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694885442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694885444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694885447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694885461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,529 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/54c4359e58f943719cdf2e6085df0495 2023-03-24 21:53:45,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694885577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694885578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,613 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=236 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/e89fa5cd38a740b693f639fbdc555a92 2023-03-24 21:53:45,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694885585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694885587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,626 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/cbc563f1bf4f49f8b14963c14d3f4940 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940 2023-03-24 21:53:45,637 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940, entries=150, sequenceid=236, filesize=11.8 K 2023-03-24 21:53:45,641 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/54c4359e58f943719cdf2e6085df0495 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495 2023-03-24 21:53:45,653 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495, entries=150, sequenceid=236, filesize=11.8 K 2023-03-24 21:53:45,656 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/e89fa5cd38a740b693f639fbdc555a92 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92 2023-03-24 21:53:45,667 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92, entries=150, sequenceid=236, filesize=11.8 K 2023-03-24 21:53:45,668 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~407.20 KB/416976, currentSize=46.96 KB/48090 for a01d0abc8722ae28834e0262613a5783 in 264ms, sequenceid=236, compaction requested=true 2023-03-24 21:53:45,668 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:45,668 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,668 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:45,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:45,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:45,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:45,679 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:45,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78ece7b2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:45,683 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:45,683 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,686 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,686 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:45,686 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:45,686 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:45,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:45,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:45,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:45,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:45,695 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:45,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:45,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:45,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:45,695 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:45,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:45,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:45,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:45,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,717 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,717 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:45,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,717 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:45,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:45,785 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:53:45,785 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:45,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:45,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:45,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:45,786 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:45,786 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:45,818 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:45,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:45,829 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/7d438bf269ec4f8eb689168853ab0ad8 2023-03-24 21:53:45,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:45,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:45,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:45,860 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:45,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:45,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:45,862 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694885854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694825849, max timestamp=1679694825855Num uniques 859; 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694825849, max timestamp=1679694825855Num uniques 859; 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694825849, max timestamp=1679694825855Num uniques 859; 2023-03-24 21:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:45,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:45,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:45,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:45,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:45,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:45,892 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:45,892 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:45,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:45,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:45,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:45,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:45,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:45,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:45,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:45,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:45,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:45,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:45,906 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/d50519d6bfba4206a3f5d71a0a0cdfa5 2023-03-24 21:53:45,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32719df9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:45,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:45,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,914 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:45,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:45,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:45,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:45,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:45,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:45,925 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:45,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:45,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:45,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:45,925 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:45,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,930 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:45,931 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:45,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:45,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:45,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:45,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:45,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:45,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:45,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:45,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:45,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:45,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:45,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:45,964 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:45,965 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:45,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:45,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:45,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d205b7e24a8149f89e3e40cf6ca78cb2 2023-03-24 21:53:45,980 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/7d438bf269ec4f8eb689168853ab0ad8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8 2023-03-24 21:53:45,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,988 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694885984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694885981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694885985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694885988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:45,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694885988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:45,998 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8, entries=150, sequenceid=247, filesize=11.8 K 2023-03-24 21:53:46,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/d50519d6bfba4206a3f5d71a0a0cdfa5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5 2023-03-24 21:53:46,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5, entries=150, sequenceid=247, filesize=11.8 K 2023-03-24 21:53:46,019 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d205b7e24a8149f89e3e40cf6ca78cb2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2 2023-03-24 21:53:46,056 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2, entries=150, sequenceid=247, filesize=11.8 K 2023-03-24 21:53:46,060 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for a01d0abc8722ae28834e0262613a5783 in 274ms, sequenceid=247, compaction requested=true 2023-03-24 21:53:46,060 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:46,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:46,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:46,063 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:46,064 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:46,064 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:46,065 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:46,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:46,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:46,080 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45650 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:46,082 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:46,083 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,083 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=44.6 K 2023-03-24 21:53:46,085 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3c867a3e2a1645158a3e1e7780007ba4, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679694824037 2023-03-24 21:53:46,085 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 75aa24a4edd84cfb9cb597c9f6229edd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679694824864 2023-03-24 21:53:46,086 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 54c4359e58f943719cdf2e6085df0495, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679694825062 2023-03-24 21:53:46,087 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d50519d6bfba4206a3f5d71a0a0cdfa5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:46,095 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55214 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:46,100 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:46,100 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,100 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=53.9 K 2023-03-24 21:53:46,101 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6d4fd6b7dd9b484fb901709b0f5186d6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679694824037 2023-03-24 21:53:46,102 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5546db00b6c340458880157e38a520b5, keycount=350, bloomtype=ROW, size=21.1 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679694824860 2023-03-24 21:53:46,104 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cbc563f1bf4f49f8b14963c14d3f4940, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679694825062 2023-03-24 21:53:46,105 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7d438bf269ec4f8eb689168853ab0ad8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:46,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694825946, max timestamp=1679694826094Num uniques 1018; 2023-03-24 21:53:46,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:46,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-24 21:53:46,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:46,110 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694825946, max timestamp=1679694826094Num uniques 1018; 2023-03-24 21:53:46,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:46,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694825946, max timestamp=1679694826094Num uniques 1018; 2023-03-24 21:53:46,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:53:46,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:46,121 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:53:46,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:46,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:53:46,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,124 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:46,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,125 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:46,125 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:46,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:46,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,136 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,138 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,138 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:46,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:46,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:46,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:46,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:46,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:46,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:46,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:46,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:46,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:46,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:46,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,185 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#13 average throughput is 0.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:46,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:46,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:46,197 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/ff4db27172024adb9dcf8563746a9ec4 2023-03-24 21:53:46,198 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#14 average throughput is 0.33 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:46,199 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:46,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:46,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df55863[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:46,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:46,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,210 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:46,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:46,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:46,211 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,211 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:46,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694826148, max timestamp=1679694826211Num uniques 859; 2023-03-24 21:53:46,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:46,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694826148, max timestamp=1679694826211Num uniques 859; 2023-03-24 21:53:46,218 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:46,220 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:46,220 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,220 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,220 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694826148, max timestamp=1679694826211Num uniques 859; 2023-03-24 21:53:46,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:46,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:46,231 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:46,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:46,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:46,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:46,231 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:46,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:46,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:46,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,261 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:46,261 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:46,261 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,314 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/72b5470bb0844731b22ab041c8e4ce0d 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694826297, max timestamp=1679694826320Num uniques 1018; 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694826297, max timestamp=1679694826320Num uniques 1018; 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694826297, max timestamp=1679694826320Num uniques 1018; 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 21:53:46,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-03-24 21:53:46,361 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:46,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:46,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:46,380 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:46,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:46,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:46,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8093, totalHeapSize=18648, min timestamp=1679694826366, max timestamp=1679694826380Num uniques 1127; 2023-03-24 21:53:46,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51603, new segment data size=8093, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:53:46,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694826366, max timestamp=1679694826380Num uniques 1127; 2023-03-24 21:53:46,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:53:46,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694826366, max timestamp=1679694826380Num uniques 1127; 2023-03-24 21:53:46,386 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:53:46,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:46,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:46,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:46,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:46,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:46,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,400 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:46,400 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:46,401 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,401 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:46,401 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:46,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:46,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d38bbd1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:46,408 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:46,408 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:46,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:46,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:46,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:46,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:46,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:46,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:46,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:46,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:46,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:46,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:46,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:46,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:46,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:46,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:46,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,437 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:46,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,437 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:46,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694826424, max timestamp=1679694826435Num uniques 1036; 2023-03-24 21:53:46,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:53:46,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694826424, max timestamp=1679694826435Num uniques 1036; 2023-03-24 21:53:46,446 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:53:46,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694826424, max timestamp=1679694826435Num uniques 1036; 2023-03-24 21:53:46,447 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:53:46,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694826479, max timestamp=1679694826489Num uniques 1145; 2023-03-24 21:53:46,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-03-24 21:53:46,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694826465, max timestamp=1679694826489Num uniques 1145; 2023-03-24 21:53:46,493 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=11220, suffix heap size=147048,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=245 2023-03-24 21:53:46,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694826479, max timestamp=1679694826489Num uniques 1145; 2023-03-24 21:53:46,494 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-03-24 21:53:46,537 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:46,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:46,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:46,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:46,551 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:46,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:46,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:46,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,557 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,557 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,557 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:46,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:46,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:46,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:46,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,561 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:46,561 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:46,561 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:46,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:46,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:46,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:46,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:46,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:46,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:46,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:46,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22f60ab3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:46,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:46,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=304, cellSize=13920, totalHeapSize=31856, min timestamp=1679694826539, max timestamp=1679694826573Num uniques 1104; 2023-03-24 21:53:46,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=13920, suffix heap size=142784,new segment heap size=31856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=304 2023-03-24 21:53:46,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:46,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:46,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:46,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:46,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694826566, max timestamp=1679694826573Num uniques 1054; 2023-03-24 21:53:46,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694826566, max timestamp=1679694826577Num uniques 1054; 2023-03-24 21:53:46,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:53:46,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:53:46,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:46,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:46,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:46,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:46,591 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:46,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:46,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:46,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:46,591 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:46,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,599 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,604 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,604 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:46,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,607 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:46,606 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:46,607 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:46,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679694826617, max timestamp=1679694826640Num uniques 1163; 2023-03-24 21:53:46,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=53258, new segment data size=12038, suffix heap size=148920,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1163, new segment cells count=263 2023-03-24 21:53:46,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679694826617, max timestamp=1679694826640Num uniques 1013; 2023-03-24 21:53:46,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-24 21:53:46,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1679694826617, max timestamp=1679694826640Num uniques 1013; 2023-03-24 21:53:46,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=12038, suffix heap size=133320,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=263 2023-03-24 21:53:46,680 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694826665, max timestamp=1679694826677Num uniques 1122; 2023-03-24 21:53:46,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694826665, max timestamp=1679694826677Num uniques 1122; 2023-03-24 21:53:46,681 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-24 21:53:46,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694826665, max timestamp=1679694826677Num uniques 1122; 2023-03-24 21:53:46,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=10162, suffix heap size=144656,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=222 2023-03-24 21:53:46,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=6870, suffix heap size=144656,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=150 2023-03-24 21:53:46,703 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/662f5e7ddc6349639835bb3a9ca1aee4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/662f5e7ddc6349639835bb3a9ca1aee4 2023-03-24 21:53:46,703 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/272d8fca6af4423b835bf4e87be54a7b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/272d8fca6af4423b835bf4e87be54a7b 2023-03-24 21:53:46,708 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:46,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:46,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:46,734 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 662f5e7ddc6349639835bb3a9ca1aee4(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:46,734 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:46,734 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=12, startTime=1679694826063; duration=0sec 2023-03-24 21:53:46,739 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 272d8fca6af4423b835bf4e87be54a7b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:46,740 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:46,740 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=12, startTime=1679694826062; duration=0sec 2023-03-24 21:53:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:46,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:46,740 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:46,740 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,741 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:46,742 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:46,742 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:46,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:46,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:46,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694826718, max timestamp=1679694826742Num uniques 1081; 2023-03-24 21:53:46,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-24 21:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694826718, max timestamp=1679694826742Num uniques 1009; 2023-03-24 21:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12866, suffix heap size=132904,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=281 2023-03-24 21:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694826718, max timestamp=1679694826742Num uniques 1081; 2023-03-24 21:53:46,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=12866, suffix heap size=140392,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=281 2023-03-24 21:53:46,748 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,748 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,750 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45650 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:46,750 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:46,750 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,750 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=44.6 K 2023-03-24 21:53:46,753 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:46,758 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4ae9a01f7e3342c0a1597b877d86cc55, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=8, earliestPutTs=1679694824037 2023-03-24 21:53:46,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=285 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/48df6a483ef04b4a9cc83e6927b8ae57 2023-03-24 21:53:46,759 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 18282588fc734896b707e2e9eae6b6d7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=204, earliestPutTs=1679694824864 2023-03-24 21:53:46,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,762 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e89fa5cd38a740b693f639fbdc555a92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=236, earliestPutTs=1679694825062 2023-03-24 21:53:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:46,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:46,764 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d205b7e24a8149f89e3e40cf6ca78cb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:46,766 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:46,766 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:46,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:46,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:46,768 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,768 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:46,768 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:46,768 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:46,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:46,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:46,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:46,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:46,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c56d541[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:46,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:46,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:46,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:46,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:46,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:46,781 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/ff4db27172024adb9dcf8563746a9ec4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4 2023-03-24 21:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:46,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:46,805 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:46,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:46,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:46,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:46,805 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:46,808 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,808 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,808 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:46,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:46,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:46,809 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:46,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694826794, max timestamp=1679694826807Num uniques 1140; 2023-03-24 21:53:46,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-24 21:53:46,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694826794, max timestamp=1679694826807Num uniques 1140; 2023-03-24 21:53:46,813 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-24 21:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694826794, max timestamp=1679694826807Num uniques 1140; 2023-03-24 21:53:46,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=8700, suffix heap size=146528,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=190 2023-03-24 21:53:46,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:46,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:46,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,820 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4, entries=200, sequenceid=285, filesize=14.3 K 2023-03-24 21:53:46,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/72b5470bb0844731b22ab041c8e4ce0d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d 2023-03-24 21:53:46,826 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#17 average throughput is 0.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:46,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:46,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:46,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:46,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:46,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:46,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:46,828 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:46,828 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:46,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:46,837 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d, entries=150, sequenceid=285, filesize=11.9 K 2023-03-24 21:53:46,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/48df6a483ef04b4a9cc83e6927b8ae57 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57 2023-03-24 21:53:46,878 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57, entries=150, sequenceid=285, filesize=11.9 K 2023-03-24 21:53:46,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=93.93 KB/96180 for a01d0abc8722ae28834e0262613a5783 in 771ms, sequenceid=285, compaction requested=false 2023-03-24 21:53:46,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:46,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:46,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=100.63 KB heapSize=282.47 KB 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:46,913 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:46,928 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:46,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:46,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:46,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:46,938 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:46,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:46,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:46,964 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,964 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e43b8510d842428bb43c5edd92ba206d 2023-03-24 21:53:46,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:46,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:46,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:46,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:46,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:46,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:46,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:46,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:46,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:46,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:46,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:46,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:46,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:46,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:46,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694886973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694886978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694886981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,000 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,000 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694886989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,003 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,003 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,003 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,003 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694886989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,041 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,042 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b17b0e8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,053 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,053 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,053 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,053 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,061 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,061 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,074 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,093 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6c375d745e3d49049b3c12a844f50001 2023-03-24 21:53:47,100 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,100 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,100 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,101 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,101 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:47,100 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694887094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694887110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38312 deadline: 1679694887110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694887110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694887110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,204 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:47,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:47,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/7e6a6642b0994b2fa4c44cc4a8af601e 2023-03-24 21:53:47,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:47,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:47,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e43b8510d842428bb43c5edd92ba206d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d 2023-03-24 21:53:47,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:47,226 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:47,226 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:47,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:47,228 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,228 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:47,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:47,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:47,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:47,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:47,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:47,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,245 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13897868[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,253 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,253 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d, entries=150, sequenceid=476, filesize=11.9 K 2023-03-24 21:53:47,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,263 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6c375d745e3d49049b3c12a844f50001 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001 2023-03-24 21:53:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,279 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/915cfffb745b4d70bb9e33cde1383b6e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/915cfffb745b4d70bb9e33cde1383b6e 2023-03-24 21:53:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,282 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,283 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,283 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:47,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:47,288 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001, entries=150, sequenceid=476, filesize=11.9 K 2023-03-24 21:53:47,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/7e6a6642b0994b2fa4c44cc4a8af601e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e 2023-03-24 21:53:47,317 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,317 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,320 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 915cfffb745b4d70bb9e33cde1383b6e(size=11.9 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:47,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694887308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,321 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:47,321 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=12, startTime=1679694826063; duration=0sec 2023-03-24 21:53:47,321 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:47,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694887314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,321 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694887317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694887321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,326 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e, entries=150, sequenceid=476, filesize=11.9 K 2023-03-24 21:53:47,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~342.19 KB/350400, currentSize=67.09 KB/68700 for a01d0abc8722ae28834e0262613a5783 in 421ms, sequenceid=476, compaction requested=true 2023-03-24 21:53:47,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:47,327 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:47,327 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:47,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:47,329 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:47,329 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:47,331 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:47,332 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:47,332 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,332 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/662f5e7ddc6349639835bb3a9ca1aee4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=35.7 K 2023-03-24 21:53:47,333 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38986 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:47,333 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 662f5e7ddc6349639835bb3a9ca1aee4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:47,336 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:47,336 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,336 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/272d8fca6af4423b835bf4e87be54a7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=38.1 K 2023-03-24 21:53:47,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:47,342 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 72b5470bb0844731b22ab041c8e4ce0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679694825982 2023-03-24 21:53:47,345 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6c375d745e3d49049b3c12a844f50001, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:47,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:53:47,344 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 272d8fca6af4423b835bf4e87be54a7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:47,347 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:47,348 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ff4db27172024adb9dcf8563746a9ec4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679694825946 2023-03-24 21:53:47,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e43b8510d842428bb43c5edd92ba206d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:47,381 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#22 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:47,383 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:47,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:47,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:47,389 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#23 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:47,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:47,391 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:47,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:47,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:47,401 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,401 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:47,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:47,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:47,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:47,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:47,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:47,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1385b38[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,457 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6ca83a8f0e3447fe8f56caa977cd944f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6ca83a8f0e3447fe8f56caa977cd944f 2023-03-24 21:53:47,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,458 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,458 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,465 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,465 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,470 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:53:47,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,471 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 6ca83a8f0e3447fe8f56caa977cd944f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:47,471 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:47,471 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=13, startTime=1679694827327; duration=0sec 2023-03-24 21:53:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,472 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:47,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,472 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,475 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:47,475 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:47,475 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,475 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/915cfffb745b4d70bb9e33cde1383b6e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=35.7 K 2023-03-24 21:53:47,476 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 915cfffb745b4d70bb9e33cde1383b6e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1679694825435 2023-03-24 21:53:47,477 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:47,477 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 48df6a483ef04b4a9cc83e6927b8ae57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=285, earliestPutTs=1679694825982 2023-03-24 21:53:47,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,477 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,481 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,481 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7e6a6642b0994b2fa4c44cc4a8af601e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:47,481 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:47,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,496 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#24 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:47,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694887519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,568 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/5c4dafcf0524417491db0e36ea2dd715 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5c4dafcf0524417491db0e36ea2dd715 2023-03-24 21:53:47,579 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 5c4dafcf0524417491db0e36ea2dd715(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:47,580 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:47,580 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=13, startTime=1679694827329; duration=0sec 2023-03-24 21:53:47,580 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:47,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:47,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:47,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:47,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:47,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:47,601 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:47,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:47,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:47,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,605 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,605 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,605 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:47,614 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:47,615 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:47,615 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:47,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,620 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,620 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,620 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,620 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,621 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-24 21:53:47,624 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-24 21:53:47,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,627 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,628 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694887624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694887625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694887626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bee6a83[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694887630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694887633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,633 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,634 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,634 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,636 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,644 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,644 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,650 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,650 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,650 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,651 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,659 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,659 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,661 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,661 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:47,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:47,761 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:47,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:47,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:47,768 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:47,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:47,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:47,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,775 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:47,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:47,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:47,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:47,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:47,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:47,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:47,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,780 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d281fe6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,787 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,789 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,794 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,794 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,802 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:47,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,807 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,807 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:47,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/ab78455cd89f496ca77069baec7c191a 2023-03-24 21:53:47,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:47,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694887852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,866 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/970aaa552d4d4c84a833f7e18dc6171e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/970aaa552d4d4c84a833f7e18dc6171e 2023-03-24 21:53:47,878 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 970aaa552d4d4c84a833f7e18dc6171e(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:47,878 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:47,878 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=13, startTime=1679694827327; duration=0sec 2023-03-24 21:53:47,879 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:47,907 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:47,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:47,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:47,918 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:47,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:47,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:47,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:47,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:47,945 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:47,945 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:47,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:47,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:47,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:47,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:47,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:47,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:47,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:47,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:47,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:47,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:47,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:47,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:47,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,959 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,959 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:47,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ba7213[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:47,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:47,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,962 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:47,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:47,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:47,967 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:47,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:47,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:47,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:47,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:47,978 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:47,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:47,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:47,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:47,978 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:47,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:47,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:47,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:47,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:47,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:47,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:47,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:47,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:47,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:47,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:47,992 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:47,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:47,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:47,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:47,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:47,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:47,994 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:47,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:47,994 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,018 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 21:53:48,019 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-24 21:53:48,094 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,106 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,120 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,120 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:48,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:48,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:48,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:48,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:48,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,128 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:48,128 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,133 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694888133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:48,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66180c07[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:48,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:48,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694888133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:48,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:48,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:48,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:48,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:48,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:48,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694888134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694888135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:48,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:48,150 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:48,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:48,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:48,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:48,150 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:48,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,166 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694888165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:48,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:48,172 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:48,172 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,272 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,278 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,284 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,285 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,285 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,285 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:48,286 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:48,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:48,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:48,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:48,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:48,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e7ac835[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:48,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:48,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:48,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:48,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:48,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:48,295 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:48,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:48,304 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:48,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:48,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:48,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:48,304 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:48,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,312 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:48,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:48,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,320 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:48,320 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,370 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c1ab097b7cb846e8b77c065aadea862e 2023-03-24 21:53:48,420 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,430 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,431 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,431 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,443 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,443 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,443 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:48,443 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:48,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:48,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:48,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3102733[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:48,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:48,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:48,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:48,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:48,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:48,454 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:48,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:48,464 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:48,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:48,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:48,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:48,464 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:48,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,473 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:48,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:48,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:48,473 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,474 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:48,474 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,574 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,581 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:48,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:48,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:48,597 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:48,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cce7a8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:48,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:48,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,602 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:48,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:48,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:48,604 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,604 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,604 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,604 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:48,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:48,609 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:48,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:48,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:48,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:48,609 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:48,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:48,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:48,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,622 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,624 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:48,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:48,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694888672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,728 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,739 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,739 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,739 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:48,739 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:48,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:48,742 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,743 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:48,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5312b522[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:48,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:48,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,747 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,747 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:48,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:48,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:48,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:48,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:48,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:48,760 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:48,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:48,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:48,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:48,760 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:48,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:48,765 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:48,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:48,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:48,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:48,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:48,774 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:48,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:48,774 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:48,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:48,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=492 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/f829e224397a4909abd64e3a5f7c2e98 2023-03-24 21:53:48,812 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/ab78455cd89f496ca77069baec7c191a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a 2023-03-24 21:53:48,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a, entries=150, sequenceid=492, filesize=11.9 K 2023-03-24 21:53:48,831 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c1ab097b7cb846e8b77c065aadea862e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e 2023-03-24 21:53:48,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e, entries=150, sequenceid=492, filesize=11.9 K 2023-03-24 21:53:48,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/f829e224397a4909abd64e3a5f7c2e98 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98 2023-03-24 21:53:48,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98, entries=150, sequenceid=492, filesize=11.9 K 2023-03-24 21:53:48,875 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a01d0abc8722ae28834e0262613a5783 in 1529ms, sequenceid=492, compaction requested=false 2023-03-24 21:53:48,875 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:48,875 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:48,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:48,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:48,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:48,892 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:48,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,895 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:48,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:48,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:48,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:48,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:48,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:48,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:48,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:48,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:48,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:48,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:48,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:48,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:48,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:48,934 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:53:48,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:48,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:48,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:48,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:48,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:48,936 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:48,996 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c 2023-03-24 21:53:49,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:49,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:49,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694889175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694889209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38322 deadline: 1679694889209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,214 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38300 deadline: 1679694889209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,315 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694889313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,322 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694889315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,323 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694889316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694889315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,458 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/919451269ddb4283ae37bbf95978b9f9 2023-03-24 21:53:49,504 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=512 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/bf0d1705e9854a669c8fa6ea8b8a4c17 2023-03-24 21:53:49,514 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c 2023-03-24 21:53:49,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694889517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,527 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c, entries=150, sequenceid=512, filesize=11.9 K 2023-03-24 21:53:49,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694889525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,529 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694889526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694889526, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/919451269ddb4283ae37bbf95978b9f9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9 2023-03-24 21:53:49,547 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9, entries=150, sequenceid=512, filesize=11.9 K 2023-03-24 21:53:49,554 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/bf0d1705e9854a669c8fa6ea8b8a4c17 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17 2023-03-24 21:53:49,562 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17, entries=150, sequenceid=512, filesize=11.9 K 2023-03-24 21:53:49,565 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for a01d0abc8722ae28834e0262613a5783 in 631ms, sequenceid=512, compaction requested=true 2023-03-24 21:53:49,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:49,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:49,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:49,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:49,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:49,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:49,574 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:49,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b41e415[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:49,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:49,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:49,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:49,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:49,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:49,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:49,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:49,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:49,597 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:49,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:49,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:49,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:49,598 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:49,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:49,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:49,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:49,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:49,606 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:49,609 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:49,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:49,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:49,688 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:53:49,688 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:49,689 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,709 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:49,710 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/14b207781cc34dc988403823f720bb96 2023-03-24 21:53:49,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:49,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:49,722 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:49,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:49,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:49,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:49,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:49,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:49,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:49,726 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,726 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:49,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:49,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:49,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:49,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:49,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:49,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:49,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:49,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:49,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:49,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:49,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:49,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:49,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39ea75f1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:49,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:49,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:49,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:49,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:49,754 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,754 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:49,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:49,755 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694889752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c1a3b3ab60f0424692c0713c4fa9fd9e 2023-03-24 21:53:49,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:49,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:49,768 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:49,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:49,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:49,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:49,768 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:49,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,773 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:49,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:49,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:49,784 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:49,784 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:49,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:49,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=529 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/24c97ba657424bb4b47bb606ec783c69 2023-03-24 21:53:49,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/14b207781cc34dc988403823f720bb96 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96 2023-03-24 21:53:49,818 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96, entries=200, sequenceid=529, filesize=14.3 K 2023-03-24 21:53:49,821 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c1a3b3ab60f0424692c0713c4fa9fd9e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e 2023-03-24 21:53:49,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694889821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694889832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694889832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694889833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e, entries=150, sequenceid=529, filesize=11.9 K 2023-03-24 21:53:49,838 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/24c97ba657424bb4b47bb606ec783c69 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69 2023-03-24 21:53:49,845 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69, entries=150, sequenceid=529, filesize=11.9 K 2023-03-24 21:53:49,846 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a01d0abc8722ae28834e0262613a5783 in 158ms, sequenceid=529, compaction requested=true 2023-03-24 21:53:49,846 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:49,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:49,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:49,847 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:49,847 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:53:49,849 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:49,853 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:49,853 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:49,853 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,854 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/970aaa552d4d4c84a833f7e18dc6171e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=50.2 K 2023-03-24 21:53:49,854 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:49,854 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:49,854 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,854 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6ca83a8f0e3447fe8f56caa977cd944f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=47.8 K 2023-03-24 21:53:49,854 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 970aaa552d4d4c84a833f7e18dc6171e, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:49,856 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6ca83a8f0e3447fe8f56caa977cd944f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:49,858 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ab78455cd89f496ca77069baec7c191a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679694826975 2023-03-24 21:53:49,858 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c1ab097b7cb846e8b77c065aadea862e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679694826975 2023-03-24 21:53:49,868 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 919451269ddb4283ae37bbf95978b9f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679694827469 2023-03-24 21:53:49,869 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c1a3b3ab60f0424692c0713c4fa9fd9e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:49,873 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 87dfc7d43ba94e7e8ab8f6f2e6cfa60c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679694827469 2023-03-24 21:53:49,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 14b207781cc34dc988403823f720bb96, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:49,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:49,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:49,878 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:49,884 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:49,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:49,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:49,901 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:49,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:49,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:49,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:49,902 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,902 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:49,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:49,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:49,921 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#34 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:49,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:49,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:49,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:49,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:49,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:49,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/dc4eced99234410babac91cd2267f3d0 2023-03-24 21:53:49,957 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:49,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:49,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:49,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:49,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:49,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:49,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b8934c3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:49,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:49,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:49,971 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:49,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:49,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:49,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:49,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:49,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:49,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694889976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:49,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:49,982 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:49,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:49,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:49,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:49,982 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:49,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:49,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:49,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,986 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:49,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:49,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:49,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:49,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:49,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:49,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:49,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:49,994 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:49,994 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:49,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:49,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,025 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e043a22d253c40dc8a69d945ac658f54 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e043a22d253c40dc8a69d945ac658f54 2023-03-24 21:53:50,036 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into e043a22d253c40dc8a69d945ac658f54(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:50,036 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:50,036 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=12, startTime=1679694829847; duration=0sec 2023-03-24 21:53:50,036 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:50,036 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:50,046 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:50,046 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:50,046 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,047 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5c4dafcf0524417491db0e36ea2dd715, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=47.8 K 2023-03-24 21:53:50,048 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5c4dafcf0524417491db0e36ea2dd715, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1679694826868 2023-03-24 21:53:50,049 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f829e224397a4909abd64e3a5f7c2e98, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=492, earliestPutTs=1679694826975 2023-03-24 21:53:50,049 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bf0d1705e9854a669c8fa6ea8b8a4c17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=512, earliestPutTs=1679694827469 2023-03-24 21:53:50,050 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 24c97ba657424bb4b47bb606ec783c69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:50,058 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/3bd86db0e4a54b5692c917a70737f6b1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3bd86db0e4a54b5692c917a70737f6b1 2023-03-24 21:53:50,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/59fbfad3bb99413db7fd989307b0ddbe 2023-03-24 21:53:50,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694890080, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,085 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:50,086 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 3bd86db0e4a54b5692c917a70737f6b1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:50,086 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:50,086 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=12, startTime=1679694829847; duration=0sec 2023-03-24 21:53:50,086 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:50,094 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,119 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:50,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,139 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,140 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,140 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,140 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,141 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,141 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,141 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:50,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:50,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:50,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:50,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,147 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,147 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:50,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6df42f35[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:50,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:50,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:50,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:50,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:50,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:50,156 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:50,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:50,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:50,163 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:50,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:50,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:50,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:50,163 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:50,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:50,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:50,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,175 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:50,175 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:50,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/5b66ba1743ca40489eeacc01b7dfee21 2023-03-24 21:53:50,208 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/dc4eced99234410babac91cd2267f3d0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0 2023-03-24 21:53:50,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0, entries=200, sequenceid=545, filesize=14.3 K 2023-03-24 21:53:50,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/59fbfad3bb99413db7fd989307b0ddbe as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe 2023-03-24 21:53:50,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 21:53:50,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/5b66ba1743ca40489eeacc01b7dfee21 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21 2023-03-24 21:53:50,248 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 21:53:50,249 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for a01d0abc8722ae28834e0262613a5783 in 372ms, sequenceid=545, compaction requested=false 2023-03-24 21:53:50,249 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:50,280 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,288 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:50,288 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:50,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:50,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,299 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:50,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:50,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:50,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:50,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/af9dfd7ddbc644f09ecbff07e99c1512 2023-03-24 21:53:50,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,323 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:50,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:50,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59a11fd2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:50,330 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:50,330 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,334 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:50,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:50,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:50,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:50,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,340 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694890337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694890338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:50,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:50,343 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:50,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:50,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:50,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:50,344 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:50,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,347 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694890341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694890341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,348 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694890341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,351 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,359 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,359 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,359 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:50,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:50,359 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,359 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,360 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:50,360 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:50,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694890444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694890449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694890452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,460 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,472 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,476 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:50,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,479 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,479 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,479 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,479 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:50,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:50,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@330ebcea[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:50,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:50,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:50,488 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:50,488 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:50,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:50,489 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:50,493 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:50,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:50,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:50,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:50,494 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:50,496 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,496 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,496 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,501 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:50,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:50,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:50,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,502 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,502 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:50,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,585 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/4e482a349a744a6781b734c5dd9ace21 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4e482a349a744a6781b734c5dd9ace21 2023-03-24 21:53:50,597 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 4e482a349a744a6781b734c5dd9ace21(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:50,598 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:50,598 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=12, startTime=1679694829847; duration=0sec 2023-03-24 21:53:50,598 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:50,602 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,609 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,613 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:50,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:50,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:50,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:50,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:50,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5143b964[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:50,623 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:50,623 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:50,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:50,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:50,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:50,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:50,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:50,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:50,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:50,632 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:50,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:50,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:50,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:50,632 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:50,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,641 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,644 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:50,645 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,648 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694890648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38276 deadline: 1679694890653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:50,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694890657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,745 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,758 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,760 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/55baa1eba81e4b0ead59bec9bbd0a898 2023-03-24 21:53:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:50,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:50,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,792 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d82b9fe[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:50,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:50,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,804 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,805 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:50,805 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:50,805 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:50,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:50,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:50,810 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,810 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:50,818 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:50,825 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:50,826 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:50,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:50,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:50,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:50,826 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:50,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:50,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:50,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:50,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:50,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:50,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:50,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:50,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:50,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:50,841 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:50,841 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:50,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:50,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:50,853 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=564 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/1db5db2f9c5845068d04ce143d6ed7c7 2023-03-24 21:53:50,864 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/af9dfd7ddbc644f09ecbff07e99c1512 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512 2023-03-24 21:53:50,873 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512, entries=200, sequenceid=564, filesize=14.3 K 2023-03-24 21:53:50,874 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/55baa1eba81e4b0ead59bec9bbd0a898 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898 2023-03-24 21:53:50,914 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898, entries=150, sequenceid=564, filesize=11.9 K 2023-03-24 21:53:50,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/1db5db2f9c5845068d04ce143d6ed7c7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7 2023-03-24 21:53:50,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7, entries=150, sequenceid=564, filesize=11.9 K 2023-03-24 21:53:50,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for a01d0abc8722ae28834e0262613a5783 in 637ms, sequenceid=564, compaction requested=true 2023-03-24 21:53:50,924 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:50,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:50,924 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:50,925 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:50,925 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:50,925 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:50,926 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:50,926 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41814 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:50,926 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:50,926 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:50,928 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,928 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3bd86db0e4a54b5692c917a70737f6b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.1 K 2023-03-24 21:53:50,928 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,929 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e043a22d253c40dc8a69d945ac658f54, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=40.8 K 2023-03-24 21:53:50,929 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3bd86db0e4a54b5692c917a70737f6b1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:50,934 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 59fbfad3bb99413db7fd989307b0ddbe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679694829745 2023-03-24 21:53:50,934 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e043a22d253c40dc8a69d945ac658f54, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:50,935 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting dc4eced99234410babac91cd2267f3d0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679694829698 2023-03-24 21:53:50,939 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting af9dfd7ddbc644f09ecbff07e99c1512, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:50,939 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 55baa1eba81e4b0ead59bec9bbd0a898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:50,941 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:50,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:50,953 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#42 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:50,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:50,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:50,958 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:50,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:50,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:50,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,961 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:50,961 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:50,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:50,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:50,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:50,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:50,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:50,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:50,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:50,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:50,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:50,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:50,972 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:50,972 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:50,972 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:53:50,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:50,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:50,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,014 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/d5a220eb153a494ea8139d39e3e1a67c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/d5a220eb153a494ea8139d39e3e1a67c 2023-03-24 21:53:51,034 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into d5a220eb153a494ea8139d39e3e1a67c(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,034 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,034 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=13, startTime=1679694830924; duration=0sec 2023-03-24 21:53:51,034 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:51,035 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:51,040 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36934 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:51,040 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:51,040 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,040 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4e482a349a744a6781b734c5dd9ace21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.1 K 2023-03-24 21:53:51,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:38324 deadline: 1679694891023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,042 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4e482a349a744a6781b734c5dd9ace21, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=529, earliestPutTs=1679694829165 2023-03-24 21:53:51,049 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/c62de5488327455bbb52195fe9f67bcd 2023-03-24 21:53:51,045 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/4dffb72c7eb141119c40dc1a6fac7b21 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/4dffb72c7eb141119c40dc1a6fac7b21 2023-03-24 21:53:51,050 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5b66ba1743ca40489eeacc01b7dfee21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=545, earliestPutTs=1679694829745 2023-03-24 21:53:51,050 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1db5db2f9c5845068d04ce143d6ed7c7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:51,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,069 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 4dffb72c7eb141119c40dc1a6fac7b21(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,070 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,070 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=13, startTime=1679694830924; duration=0sec 2023-03-24 21:53:51,070 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:51,084 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#46 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:51,137 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/0bc391bb3fdf46e2af3fc22904ef113a 2023-03-24 21:53:51,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694891143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694891348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694891349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694891354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,522 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/27574796d44e437c8d810b70693c18b1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/27574796d44e437c8d810b70693c18b1 2023-03-24 21:53:51,529 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 27574796d44e437c8d810b70693c18b1(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,529 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,529 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=13, startTime=1679694830925; duration=0sec 2023-03-24 21:53:51,529 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:51,588 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=583 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/62e8f004855b4e759002dbb8a93945cc 2023-03-24 21:53:51,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/c62de5488327455bbb52195fe9f67bcd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd 2023-03-24 21:53:51,602 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 21:53:51,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/0bc391bb3fdf46e2af3fc22904ef113a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a 2023-03-24 21:53:51,616 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 21:53:51,617 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/62e8f004855b4e759002dbb8a93945cc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc 2023-03-24 21:53:51,632 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc, entries=150, sequenceid=583, filesize=11.9 K 2023-03-24 21:53:51,633 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for a01d0abc8722ae28834e0262613a5783 in 661ms, sequenceid=583, compaction requested=false 2023-03-24 21:53:51,633 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,633 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:51,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:51,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:51,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:51,645 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,645 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:51,651 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c0c6a13[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:51,651 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:51,651 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:51,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:51,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:51,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:51,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,657 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,657 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:51,657 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:51,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:51,658 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:51,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,658 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:51,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:51,667 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:51,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:51,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:51,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:51,667 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:51,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,677 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:51,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:51,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,679 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:51,679 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:51,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/706fb5c45b16474185326c7384940321 2023-03-24 21:53:51,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694891691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/9f54bee61e5f40df81983b5e6ec8a0aa 2023-03-24 21:53:51,733 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/3ddaf1ec735b41a79a549021b0bdc323 2023-03-24 21:53:51,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/706fb5c45b16474185326c7384940321 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321 2023-03-24 21:53:51,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321, entries=150, sequenceid=601, filesize=11.9 K 2023-03-24 21:53:51,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/9f54bee61e5f40df81983b5e6ec8a0aa as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa 2023-03-24 21:53:51,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa, entries=150, sequenceid=601, filesize=11.9 K 2023-03-24 21:53:51,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/3ddaf1ec735b41a79a549021b0bdc323 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323 2023-03-24 21:53:51,766 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323, entries=150, sequenceid=601, filesize=11.9 K 2023-03-24 21:53:51,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for a01d0abc8722ae28834e0262613a5783 in 110ms, sequenceid=601, compaction requested=true 2023-03-24 21:53:51,767 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:51,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:51,768 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:51,768 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:51,768 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:51,769 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:51,769 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:51,769 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:51,769 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:51,769 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,769 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,769 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/d5a220eb153a494ea8139d39e3e1a67c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.2 K 2023-03-24 21:53:51,769 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/4dffb72c7eb141119c40dc1a6fac7b21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.2 K 2023-03-24 21:53:51,770 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d5a220eb153a494ea8139d39e3e1a67c, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:51,771 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4dffb72c7eb141119c40dc1a6fac7b21, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:51,772 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c62de5488327455bbb52195fe9f67bcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679694830339 2023-03-24 21:53:51,772 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0bc391bb3fdf46e2af3fc22904ef113a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679694830339 2023-03-24 21:53:51,772 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 706fb5c45b16474185326c7384940321, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:51,773 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9f54bee61e5f40df81983b5e6ec8a0aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:51,779 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:51,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:51,781 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#51 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:51,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:51,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:51,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:51,784 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:51,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:51,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:51,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:51,784 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#52 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:51,785 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,785 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,788 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:51,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:51,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:51,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:51,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:51,804 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:51,804 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:51,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:51,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:51,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:51,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:51,810 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:51,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:51,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:51,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:51,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:51,811 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:51,812 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,812 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:51,812 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,812 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:51,812 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:51,818 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,818 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:51,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa37694[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:51,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:51,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:51,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:51,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:51,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:51,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,827 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:51,827 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,827 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:51,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:51,831 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:51,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:51,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:51,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:51,831 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:51,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,834 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,837 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/77c43b8e793748d3862e6e2ab66648b0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/77c43b8e793748d3862e6e2ab66648b0 2023-03-24 21:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694891835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,844 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:51,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:51,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:51,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:51,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,844 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:51,844 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,846 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 77c43b8e793748d3862e6e2ab66648b0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,846 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,846 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=13, startTime=1679694831768; duration=0sec 2023-03-24 21:53:51,846 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:51,846 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:51,849 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:51,849 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:51,849 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,850 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/27574796d44e437c8d810b70693c18b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.2 K 2023-03-24 21:53:51,853 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 27574796d44e437c8d810b70693c18b1, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=564, earliestPutTs=1679694829955 2023-03-24 21:53:51,856 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 62e8f004855b4e759002dbb8a93945cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=583, earliestPutTs=1679694830339 2023-03-24 21:53:51,857 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 3ddaf1ec735b41a79a549021b0bdc323, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:51,864 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c655cd375170400e9d39e1bae8eab599 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c655cd375170400e9d39e1bae8eab599 2023-03-24 21:53:51,868 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#54 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:51,883 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into c655cd375170400e9d39e1bae8eab599(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,883 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,883 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=13, startTime=1679694831768; duration=0sec 2023-03-24 21:53:51,883 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:51,902 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/0c9663e55a1c4a98b6b65d7e91b99de0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0c9663e55a1c4a98b6b65d7e91b99de0 2023-03-24 21:53:51,910 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 0c9663e55a1c4a98b6b65d7e91b99de0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:51,910 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:51,910 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=13, startTime=1679694831768; duration=0sec 2023-03-24 21:53:51,910 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:51,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694891938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,945 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:51,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694891944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:51,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:51,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694891944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:51,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:51,949 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:51,949 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:51,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:51,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:51,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:51,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:51,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:51,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:51,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:51,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:51,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:51,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:51,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:51,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:51,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:51,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:51,956 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,956 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:51,956 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:51,957 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:51,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:51,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:51,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:51,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:51,960 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:51,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4667763d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:51,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:51,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,967 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:51,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:51,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:51,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:51,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:51,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:51,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:51,974 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:51,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:51,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:51,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:51,974 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:51,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:51,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:51,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:51,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:51,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:51,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:51,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:51,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:51,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:51,993 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:51,993 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,107 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,119 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,119 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,119 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,119 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e226a46[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,126 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,133 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,134 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,140 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,141 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694892145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694892147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694892149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,159 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,160 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,160 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,260 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,270 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,271 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,271 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,271 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/3e83ccef651a4b61b09ccbf48e977f04 2023-03-24 21:53:52,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,286 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,289 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,289 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,289 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,289 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c9f56563113b4c44a3c2aff6596710bd 2023-03-24 21:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d1dd63[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,312 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,312 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=621 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/3c956d25e71246d6bb3ce8a5bdae4124 2023-03-24 21:53:52,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:52,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,329 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,330 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,330 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,341 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/3e83ccef651a4b61b09ccbf48e977f04 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04 2023-03-24 21:53:52,348 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04, entries=300, sequenceid=621, filesize=19.0 K 2023-03-24 21:53:52,350 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/c9f56563113b4c44a3c2aff6596710bd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd 2023-03-24 21:53:52,366 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd, entries=150, sequenceid=621, filesize=11.9 K 2023-03-24 21:53:52,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/3c956d25e71246d6bb3ce8a5bdae4124 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124 2023-03-24 21:53:52,392 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124, entries=150, sequenceid=621, filesize=11.9 K 2023-03-24 21:53:52,393 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for a01d0abc8722ae28834e0262613a5783 in 583ms, sequenceid=621, compaction requested=false 2023-03-24 21:53:52,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:52,430 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,441 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:52,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 21:53:52,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:52,452 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:52,456 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,456 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,476 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,476 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,476 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694892489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,496 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694892490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,497 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,497 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,497 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694892491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7057101b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,501 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,501 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,501 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,505 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,511 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,511 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,523 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,528 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,528 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/062eb8c3bb8746a4ab5cd978a1986d3c 2023-03-24 21:53:52,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694892594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694892598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694892599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,628 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,632 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,635 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,637 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,637 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,637 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,637 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,637 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d7c46b2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,645 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,645 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,651 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,655 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,655 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,658 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,659 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,664 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,664 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,664 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,664 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,764 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,771 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,774 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,777 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,777 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,777 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,777 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,781 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50d16d01[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,781 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,786 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,786 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,789 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,789 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,789 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,790 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,795 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:52,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,796 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,800 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694892800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694892804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:52,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694892804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,896 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:52,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:52,900 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:52,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:52,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:52,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,902 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:52,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:52,904 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:52,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:52,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:52,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:52,905 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,905 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:52,905 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:52,905 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:52,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:52,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:52,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:52,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f0488bd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:52,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:52,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:52,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:52,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:52,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:52,916 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,917 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:52,917 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,917 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:52,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:52,920 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:52,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:52,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:52,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:52,920 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:52,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:52,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:52,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:52,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,928 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:52,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:52,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:52,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:52,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:52,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:52,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:52,937 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:52,938 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:52,938 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:52,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/aa70253a2c9c41b880f9ebf2c2affcb0 2023-03-24 21:53:53,038 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:53,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:53,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:53,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:53,045 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:53,045 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,045 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:53,045 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:53,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:53,051 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:53,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:53,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:53,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:53,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:53,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:53,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:53,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:53,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:53,054 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:53,054 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:53,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:53,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,057 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,057 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:53,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:53,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@779b908e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:53,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:53,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:53,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:53,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:53,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:53,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:53,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:53,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:53,068 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:53,068 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:53,068 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:53,068 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:53,068 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:53,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,096 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:53,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:53,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,105 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,105 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:53,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:53,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,106 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694893104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,110 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694893109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694893110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,205 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:53,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:53,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:53,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:53,209 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:53,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:53,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:53,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,211 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,211 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:53,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:53,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:53,213 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,213 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:53,213 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:53,213 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,213 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:53,214 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:53,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:53,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@183d0119[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:53,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:53,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,218 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:53,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:53,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:53,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:53,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:53,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:53,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:53,224 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:53,224 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:53,224 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:53,224 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:53,224 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:53,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,228 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:53,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:53,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,231 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:53,231 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:53,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:53,331 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:53,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:53,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:53,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:53,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:53,335 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:53,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:53,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:53,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,338 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:53,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:53,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:53,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:53,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:53,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:53,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:53,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:53,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:53,343 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:53,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:53,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43dfce7a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:53,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,345 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,346 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:53,346 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:53,346 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:53,346 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,346 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:53,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:53,347 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:53,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:53,350 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:53,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:53,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:53,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:53,350 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:53,353 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,353 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,354 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,354 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,357 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,357 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,358 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:53,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,358 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:53,358 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:53,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:53,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694893359, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,362 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:53,365 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694893362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,366 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:53,405 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=637 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/50c2f3447be147cc81f7253a8026f88d 2023-03-24 21:53:53,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/062eb8c3bb8746a4ab5cd978a1986d3c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c 2023-03-24 21:53:53,417 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c, entries=150, sequenceid=637, filesize=11.9 K 2023-03-24 21:53:53,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/aa70253a2c9c41b880f9ebf2c2affcb0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0 2023-03-24 21:53:53,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0, entries=100, sequenceid=637, filesize=9.5 K 2023-03-24 21:53:53,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/50c2f3447be147cc81f7253a8026f88d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d 2023-03-24 21:53:53,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d, entries=100, sequenceid=637, filesize=9.5 K 2023-03-24 21:53:53,431 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for a01d0abc8722ae28834e0262613a5783 in 980ms, sequenceid=637, compaction requested=true 2023-03-24 21:53:53,431 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:53,431 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:53,431 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:53,431 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:53,432 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:53,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 44454 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:53,434 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:53,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:53,434 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:53,434 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,434 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,434 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/77c43b8e793748d3862e6e2ab66648b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=43.4 K 2023-03-24 21:53:53,434 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c655cd375170400e9d39e1bae8eab599, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=33.9 K 2023-03-24 21:53:53,435 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 77c43b8e793748d3862e6e2ab66648b0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:53,438 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c655cd375170400e9d39e1bae8eab599, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:53,438 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 3e83ccef651a4b61b09ccbf48e977f04, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679694831687 2023-03-24 21:53:53,438 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c9f56563113b4c44a3c2aff6596710bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679694831692 2023-03-24 21:53:53,438 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 062eb8c3bb8746a4ab5cd978a1986d3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831829 2023-03-24 21:53:53,439 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting aa70253a2c9c41b880f9ebf2c2affcb0, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831829 2023-03-24 21:53:53,449 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#60 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:53,450 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#61 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:53,459 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:53,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:53,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:53,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:53,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:53,465 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:53,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:53,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:53,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,467 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:53,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:53,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:53,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:53,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:53,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:53,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:53,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:53,470 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:53,470 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:53,473 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/924a890a41c7428d9c17b13846982e5c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/924a890a41c7428d9c17b13846982e5c 2023-03-24 21:53:53,480 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 924a890a41c7428d9c17b13846982e5c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:53,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:53,480 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=13, startTime=1679694833431; duration=0sec 2023-03-24 21:53:53,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:53,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:53:53,481 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34694 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:53:53,481 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:53,482 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,482 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0c9663e55a1c4a98b6b65d7e91b99de0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=33.9 K 2023-03-24 21:53:53,482 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0c9663e55a1c4a98b6b65d7e91b99de0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1679694831018 2023-03-24 21:53:53,483 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3c956d25e71246d6bb3ce8a5bdae4124, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=621, earliestPutTs=1679694831692 2023-03-24 21:53:53,484 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 50c2f3447be147cc81f7253a8026f88d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831829 2023-03-24 21:53:53,487 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9e604678655646ec9e2c76d131a10e8b 2023-03-24 21:53:53,493 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:53,516 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/5a9b30df673f494e8007c0c67375ac76 2023-03-24 21:53:53,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:53,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:53,638 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694893633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694893634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694893634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694893740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,745 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694893744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694893745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,868 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/5b8cb6e2165f4e9bbd5d15dc1c519c25 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5b8cb6e2165f4e9bbd5d15dc1c519c25 2023-03-24 21:53:53,874 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 5b8cb6e2165f4e9bbd5d15dc1c519c25(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:53,874 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:53,874 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=13, startTime=1679694833431; duration=0sec 2023-03-24 21:53:53,874 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:53,915 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/a88e04e06dde4c2998f60675e3d84ff9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a88e04e06dde4c2998f60675e3d84ff9 2023-03-24 21:53:53,921 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into a88e04e06dde4c2998f60675e3d84ff9(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:53,921 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:53,921 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=13, startTime=1679694833432; duration=0sec 2023-03-24 21:53:53,922 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:53,928 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=656 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/0593f607e5ff4452a0e65ee4a1c23946 2023-03-24 21:53:53,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9e604678655646ec9e2c76d131a10e8b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b 2023-03-24 21:53:53,939 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b, entries=150, sequenceid=656, filesize=11.9 K 2023-03-24 21:53:53,940 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/5a9b30df673f494e8007c0c67375ac76 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76 2023-03-24 21:53:53,945 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76, entries=150, sequenceid=656, filesize=11.9 K 2023-03-24 21:53:53,946 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/0593f607e5ff4452a0e65ee4a1c23946 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946 2023-03-24 21:53:53,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694893946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694893948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:53,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694893950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,952 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946, entries=150, sequenceid=656, filesize=11.9 K 2023-03-24 21:53:53,953 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for a01d0abc8722ae28834e0262613a5783 in 483ms, sequenceid=656, compaction requested=false 2023-03-24 21:53:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:53,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:53,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:53,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:53,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:53,956 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:53,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5885f7dc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:53,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:53,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,960 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,960 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:53,960 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:53,960 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:53,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:53,961 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:53,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:53,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:53,965 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:53,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:53,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:53,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:53,965 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:53,967 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,968 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,968 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:53,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:53,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:53,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:53,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:53,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:53,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:53,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:53,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:53,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:53,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:53,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:53,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:53,979 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:53,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,979 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:53,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:53,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,079 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:54,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:54,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:54,084 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:54,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:54,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:54,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,086 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,086 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,087 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:54,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:54,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:54,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:54,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:54,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:54,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:54,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:54,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:54,088 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 21:53:54,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:54,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:54,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:54,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:54,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:54,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:54,105 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9b53a4d617ca4a1090017174e0db3b4f 2023-03-24 21:53:54,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:54,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:54,272 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694894270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694894271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694894272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694894374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694894375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694894375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,522 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/2cefed0c48884b73aac4611b429108d8 2023-03-24 21:53:54,539 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=673 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/775d93e536784a6d910994383f60cfeb 2023-03-24 21:53:54,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9b53a4d617ca4a1090017174e0db3b4f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f 2023-03-24 21:53:54,557 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 21:53:54,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/2cefed0c48884b73aac4611b429108d8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8 2023-03-24 21:53:54,567 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 21:53:54,568 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/775d93e536784a6d910994383f60cfeb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb 2023-03-24 21:53:54,574 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb, entries=150, sequenceid=673, filesize=11.9 K 2023-03-24 21:53:54,575 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for a01d0abc8722ae28834e0262613a5783 in 487ms, sequenceid=673, compaction requested=true 2023-03-24 21:53:54,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:54,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:54,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:54,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:54,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:54,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:54,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 21:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694834266, max timestamp=1679694834581Num uniques 859; 2023-03-24 21:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694834266, max timestamp=1679694834581Num uniques 859; 2023-03-24 21:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,583 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694834266, max timestamp=1679694834581Num uniques 859; 2023-03-24 21:53:54,584 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:53:54,583 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@733113a2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:54,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:54,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:54,584 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:54,586 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:54,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:54,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:54,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:54,588 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:54,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:54,592 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:54,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:54,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:54,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:54,593 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:54,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,602 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,602 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,605 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:54,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:54,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,606 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:54,606 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:54,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:54,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694834595, max timestamp=1679694834607Num uniques 859; 2023-03-24 21:53:54,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:54,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694834595, max timestamp=1679694834607Num uniques 859; 2023-03-24 21:53:54,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:54,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694834595, max timestamp=1679694834607Num uniques 859; 2023-03-24 21:53:54,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:54,616 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/1927f8457d0c49e0b67293b93f210787 2023-03-24 21:53:54,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694894652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,656 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694894653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694834624, max timestamp=1679694834655Num uniques 1168; 2023-03-24 21:53:54,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-03-24 21:53:54,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694834624, max timestamp=1679694834655Num uniques 1168; 2023-03-24 21:53:54,658 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-03-24 21:53:54,659 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694834624, max timestamp=1679694834655Num uniques 1168; 2023-03-24 21:53:54,660 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=53488, new segment data size=7688, suffix heap size=149440,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1168, new segment cells count=168 2023-03-24 21:53:54,706 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:54,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:54,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:54,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:54,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:54,711 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:54,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:54,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:54,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,714 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:54,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:54,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:54,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:54,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:54,718 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,718 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:54,718 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:54,718 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:54,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:54,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,721 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,722 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,722 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:54,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694834655, max timestamp=1679694834720Num uniques 1027; 2023-03-24 21:53:54,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 21:53:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:54,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@374e114[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:54,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:54,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694834655, max timestamp=1679694834720Num uniques 1027; 2023-03-24 21:53:54,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 21:53:54,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694834655, max timestamp=1679694834720Num uniques 1027; 2023-03-24 21:53:54,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-24 21:53:54,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,725 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:54,725 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:54,725 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:54,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:54,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:54,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:54,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:54,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:54,737 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:54,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:54,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:54,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:54,737 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:54,740 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,740 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,740 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,740 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,742 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:54,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,746 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:54,746 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:54,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679694834765, max timestamp=1679694834781Num uniques 1036; 2023-03-24 21:53:54,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-03-24 21:53:54,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679694834765, max timestamp=1679694834781Num uniques 1036; 2023-03-24 21:53:54,784 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-03-24 21:53:54,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=336, cellSize=15386, totalHeapSize=35184, min timestamp=1679694834765, max timestamp=1679694834781Num uniques 1036; 2023-03-24 21:53:54,786 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=15386, suffix heap size=135712,new segment heap size=35184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=336 2023-03-24 21:53:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694894821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694834812, max timestamp=1679694834822Num uniques 1195; 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694834812, max timestamp=1679694834822Num uniques 1195; 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694834812, max timestamp=1679694834822Num uniques 1195; 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-03-24 21:53:54,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=54730, new segment data size=8930, suffix heap size=152248,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1195, new segment cells count=195 2023-03-24 21:53:54,846 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:54,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:54,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:54,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:54,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:54,850 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:54,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:54,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:54,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:54,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:54,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:54,854 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,854 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:54,854 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:54,854 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:54,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:54,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:54,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:54,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee5531e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:54,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:54,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:54,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:54,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:54,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:54,865 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:54,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:54,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:54,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:54,866 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:54,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:54,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:54,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:54,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,872 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:54,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1679694834843, max timestamp=1679694834879Num uniques 1054; 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1679694834843, max timestamp=1679694834879Num uniques 1054; 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=354, cellSize=16210, totalHeapSize=37056, min timestamp=1679694834843, max timestamp=1679694834879Num uniques 1054; 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-03-24 21:53:54,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=16210, suffix heap size=137584,new segment heap size=37056 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=354 2023-03-24 21:53:54,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694894909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694894915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:54,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694894928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:54,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:54,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:54,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:54,977 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:54,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:54,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:54,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:54,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:54,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:54,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:54,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:54,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:54,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:54,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:54,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:54,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:54,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:54,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:54,988 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:54,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49297809[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:54,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:54,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:54,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:54,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:54,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:54,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:54,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:54,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:54,997 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:54,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:54,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:54,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:54,997 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,007 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:55,017 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694895016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694895021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/65314cb743e64ac4a71efa86bd6fe9fe 2023-03-24 21:53:55,107 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,113 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,118 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,121 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@327ece8d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,132 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,133 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,133 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694895133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,136 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,136 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,140 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,143 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,146 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,146 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694895219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694895226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,246 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a6f597[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,265 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,265 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,268 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:55,272 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,272 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,380 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,385 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,392 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,393 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,393 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,394 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,394 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,395 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cf39fe[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,398 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,408 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,408 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,409 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,411 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,415 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,415 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,415 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694895434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=694 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/4f72f3f3d053448ab33856c10a4eb68c 2023-03-24 21:53:55,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/1927f8457d0c49e0b67293b93f210787 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787 2023-03-24 21:53:55,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787, entries=150, sequenceid=694, filesize=11.9 K 2023-03-24 21:53:55,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/65314cb743e64ac4a71efa86bd6fe9fe as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe 2023-03-24 21:53:55,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe, entries=150, sequenceid=694, filesize=11.9 K 2023-03-24 21:53:55,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/4f72f3f3d053448ab33856c10a4eb68c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c 2023-03-24 21:53:55,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c, entries=150, sequenceid=694, filesize=11.9 K 2023-03-24 21:53:55,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=161.02 KB/164880 for a01d0abc8722ae28834e0262613a5783 in 897ms, sequenceid=694, compaction requested=true 2023-03-24 21:53:55,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:55,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:55,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:55,479 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:55,479 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:55,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:55,480 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:55,480 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:55,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:55,480 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:55,480 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5b8cb6e2165f4e9bbd5d15dc1c519c25, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=48.3 K 2023-03-24 21:53:55,480 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,481 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/924a890a41c7428d9c17b13846982e5c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=48.3 K 2023-03-24 21:53:55,481 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5b8cb6e2165f4e9bbd5d15dc1c519c25, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831692 2023-03-24 21:53:55,481 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 924a890a41c7428d9c17b13846982e5c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831692 2023-03-24 21:53:55,481 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9e604678655646ec9e2c76d131a10e8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1679694832476 2023-03-24 21:53:55,482 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5a9b30df673f494e8007c0c67375ac76, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1679694832476 2023-03-24 21:53:55,482 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9b53a4d617ca4a1090017174e0db3b4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679694833615 2023-03-24 21:53:55,482 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2cefed0c48884b73aac4611b429108d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679694833615 2023-03-24 21:53:55,482 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1927f8457d0c49e0b67293b93f210787, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:55,482 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 65314cb743e64ac4a71efa86bd6fe9fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:55,495 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#72 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:55,495 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#73 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:55,515 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:55,527 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=167.72 KB heapSize=467.02 KB 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:55,527 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:53:55,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,531 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,531 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,531 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,531 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,533 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,534 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694895535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694895536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,550 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,550 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,550 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,550 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,553 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,554 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,554 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a4f09e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,565 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/b551571ce1154987acf4c60349184c84 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/b551571ce1154987acf4c60349184c84 2023-03-24 21:53:55,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/6aa062eaec4f4e9ea5231e6473ac8f69 2023-03-24 21:53:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,576 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into b551571ce1154987acf4c60349184c84(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:55,577 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:55,577 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=12, startTime=1679694835479; duration=0sec 2023-03-24 21:53:55,577 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:55,577 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:53:55,575 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,579 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,580 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49426 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:53:55,580 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:55,580 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,580 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a88e04e06dde4c2998f60675e3d84ff9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=48.3 K 2023-03-24 21:53:55,582 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a88e04e06dde4c2998f60675e3d84ff9, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=637, earliestPutTs=1679694831692 2023-03-24 21:53:55,582 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0593f607e5ff4452a0e65ee4a1c23946, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=656, earliestPutTs=1679694832476 2023-03-24 21:53:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,583 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 775d93e536784a6d910994383f60cfeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=673, earliestPutTs=1679694833615 2023-03-24 21:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,584 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4f72f3f3d053448ab33856c10a4eb68c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,590 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,590 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,590 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:55,594 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:55,605 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6373d6ee9b7343729209a406f8e14d6a 2023-03-24 21:53:55,624 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/ae0e33c3f4b7469da193f49db0b245fc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ae0e33c3f4b7469da193f49db0b245fc 2023-03-24 21:53:55,634 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into ae0e33c3f4b7469da193f49db0b245fc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:55,635 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:55,635 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=12, startTime=1679694835479; duration=0sec 2023-03-24 21:53:55,635 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:55,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694895638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694895641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,690 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,695 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446a5ca1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,710 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,711 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,718 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,718 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,722 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:55,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,731 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,731 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,831 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,836 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,841 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,841 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,841 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,841 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65c103f6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,847 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,847 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,847 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694895844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,848 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694895845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,866 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:55,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:55,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,866 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:55,867 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:55,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,941 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694895939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,958 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/6474459208a54754a6f6fdc73ceb4ab5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6474459208a54754a6f6fdc73ceb4ab5 2023-03-24 21:53:55,964 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 6474459208a54754a6f6fdc73ceb4ab5(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:55,964 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:55,964 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=12, startTime=1679694835478; duration=0sec 2023-03-24 21:53:55,964 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:55,967 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:55,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:55,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:55,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:55,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:55,970 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:55,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:55,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:55,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:55,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:55,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:55,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:55,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:55,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:55,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:55,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:55,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:55,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:55,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:55,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:55,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:55,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:55,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:55,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3ac679[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:55,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:55,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:55,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:55,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:55,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:55,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:55,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:55,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:55,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:55,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:55,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:55,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:55,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:55,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:55,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:55,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:55,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:55,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:55,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:56,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:56,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:56,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:56,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,000 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:56,000 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:56,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:56,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/71d1635a677f44a8ae50c84b56986aee 2023-03-24 21:53:56,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/6aa062eaec4f4e9ea5231e6473ac8f69 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69 2023-03-24 21:53:56,060 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69, entries=200, sequenceid=818, filesize=14.3 K 2023-03-24 21:53:56,061 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6373d6ee9b7343729209a406f8e14d6a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a 2023-03-24 21:53:56,073 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a, entries=150, sequenceid=818, filesize=11.9 K 2023-03-24 21:53:56,074 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/71d1635a677f44a8ae50c84b56986aee as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee 2023-03-24 21:53:56,080 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee, entries=150, sequenceid=818, filesize=11.9 K 2023-03-24 21:53:56,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~466.97 KB/478176, currentSize=20.13 KB/20610 for a01d0abc8722ae28834e0262613a5783 in 554ms, sequenceid=818, compaction requested=false 2023-03-24 21:53:56,081 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:56,101 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:56,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:56,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:56,119 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:56,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:56,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:56,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:56,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:56,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:56,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:56,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:56,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:56,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:56,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:56,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:56,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:56,124 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:56,124 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:56,125 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:56,125 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:56,162 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:56,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694836183, max timestamp=1679694836199Num uniques 859; 2023-03-24 21:53:56,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694836183, max timestamp=1679694836199Num uniques 859; 2023-03-24 21:53:56,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:53:56,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:53:56,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694896238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,246 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694896242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694896346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694896348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694896548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694896551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,590 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/683a6ec756f3450599002aae34b4a6ae 2023-03-24 21:53:56,603 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/8a5b45d1d5fb466cb4ec8261642ce9d5 2023-03-24 21:53:56,625 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/beabe816691a48c588a6461082910776 2023-03-24 21:53:56,631 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/683a6ec756f3450599002aae34b4a6ae as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae 2023-03-24 21:53:56,640 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae, entries=50, sequenceid=827, filesize=7.1 K 2023-03-24 21:53:56,641 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/8a5b45d1d5fb466cb4ec8261642ce9d5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5 2023-03-24 21:53:56,646 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5, entries=50, sequenceid=827, filesize=7.1 K 2023-03-24 21:53:56,648 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/beabe816691a48c588a6461082910776 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776 2023-03-24 21:53:56,661 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776, entries=50, sequenceid=827, filesize=7.1 K 2023-03-24 21:53:56,662 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=161.02 KB/164880 for a01d0abc8722ae28834e0262613a5783 in 537ms, sequenceid=827, compaction requested=true 2023-03-24 21:53:56,662 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:56,662 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:56,662 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:56,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:56,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:56,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:56,664 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,664 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:56,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:56,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:56,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393df520[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:56,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:56,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,668 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:56,668 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:56,668 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:56,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:56,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:56,669 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:56,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:56,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:56,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:56,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:56,673 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:56,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:56,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:56,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:56,673 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:56,676 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,676 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,676 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,677 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:56,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:56,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:56,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:56,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,681 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:56,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:56,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:56,681 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:56,782 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:56,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:56,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:56,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:56,785 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:56,787 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:56,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:56,789 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:56,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:56,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:56,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:56,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:56,790 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:56,790 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:56,791 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=161.02 KB heapSize=463.39 KB 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:56,791 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:56,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:56,859 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:56,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694896861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694896862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694896948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694896963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:56,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:56,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694896968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694897168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,169 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694897169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,207 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/f3596ebe56d5474aa4ca834c322c11ca 2023-03-24 21:53:57,225 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/0c59c302cb6c46ea8f9aea147799867e 2023-03-24 21:53:57,267 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=71.56 KB at sequenceid=862 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/a2e1723e86364e509bce7029f20d377d 2023-03-24 21:53:57,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/f3596ebe56d5474aa4ca834c322c11ca as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca 2023-03-24 21:53:57,282 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca, entries=150, sequenceid=862, filesize=11.9 K 2023-03-24 21:53:57,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/0c59c302cb6c46ea8f9aea147799867e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e 2023-03-24 21:53:57,289 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e, entries=150, sequenceid=862, filesize=11.9 K 2023-03-24 21:53:57,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/a2e1723e86364e509bce7029f20d377d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d 2023-03-24 21:53:57,296 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d, entries=150, sequenceid=862, filesize=11.9 K 2023-03-24 21:53:57,297 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~463.34 KB/474464, currentSize=20.13 KB/20610 for a01d0abc8722ae28834e0262613a5783 in 506ms, sequenceid=862, compaction requested=true 2023-03-24 21:53:57,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:57,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:57,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:57,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:57,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:57,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,299 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f2439f2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:57,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:57,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:57,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:57,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:57,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:57,310 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:57,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:57,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:57,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:57,310 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:57,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:57,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:57,325 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:57,325 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:57,376 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:57,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:57,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694837396, max timestamp=1679694837413Num uniques 859; 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694837396, max timestamp=1679694837413Num uniques 859; 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694837396, max timestamp=1679694837413Num uniques 859; 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:57,416 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:53:57,425 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:57,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:57,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:57,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:57,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:57,428 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:57,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:57,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:57,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,431 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,431 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:57,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,431 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:57,432 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:57,432 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:57,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:57,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:57,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:57,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:57,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:57,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:57,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42975153[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:57,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:57,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:57,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:57,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:57,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:57,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:57,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:57,442 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:57,442 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:57,443 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:57,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694897442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694897445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:57,454 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:57,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:57,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:57,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694897471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694897472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,549 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694897547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694897548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:57,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:57,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:57,558 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:57,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:57,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:57,559 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,559 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,560 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:57,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:57,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:57,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:57,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:57,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:57,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:57,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:57,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:57,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:57,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:57,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:57,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ad65e76[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:57,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:57,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:57,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:57,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:57,571 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:57,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:57,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:57,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:57,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:57,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:57,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:57,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,582 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,582 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,682 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:57,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:57,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:57,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:57,686 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:57,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:57,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:57,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:57,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:57,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:57,689 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:57,689 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:57,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:57,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:57,690 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,690 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:57,690 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:57,690 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:57,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:57,692 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:57,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:57,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:57,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c143ef9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:57,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:57,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:57,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:57,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:57,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:57,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:57,700 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:57,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:57,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:57,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:57,700 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:57,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,711 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:57,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,711 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,712 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,712 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,712 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,712 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:57,712 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694897751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694897751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,800 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=872 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/c570bfc0c11d4355a958c02835bf48f0 2023-03-24 21:53:57,812 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:57,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=872 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/15747cff1a914274a6b1ba022e181141 2023-03-24 21:53:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:57,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:57,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:57,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:57,825 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:57,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:57,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:57,839 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:57,839 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,839 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:57,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:57,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:57,845 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:57,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:57,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:57,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:57,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:57,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:57,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,848 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=872 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d0ce60b079324f3cac05ecd74afb3ecf 2023-03-24 21:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,849 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,849 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:57,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f17720[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:57,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:57,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/c570bfc0c11d4355a958c02835bf48f0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0 2023-03-24 21:53:57,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:57,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:57,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:57,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:57,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:57,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:57,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0, entries=150, sequenceid=872, filesize=11.9 K 2023-03-24 21:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/15747cff1a914274a6b1ba022e181141 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141 2023-03-24 21:53:57,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:57,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:57,865 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:57,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:57,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:57,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:57,865 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:57,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:57,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:57,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,868 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141, entries=150, sequenceid=872, filesize=11.9 K 2023-03-24 21:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:57,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d0ce60b079324f3cac05ecd74afb3ecf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf 2023-03-24 21:53:57,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:57,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:57,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:57,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,872 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:57,873 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:57,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:57,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:57,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:57,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf, entries=150, sequenceid=872, filesize=11.9 K 2023-03-24 21:53:57,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a01d0abc8722ae28834e0262613a5783 in 501ms, sequenceid=872, compaction requested=true 2023-03-24 21:53:57,877 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:57,877 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:57,878 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:53:57,878 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:53:57,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:57,878 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:57,881 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59302 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:53:57,881 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:53:57,881 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,881 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6474459208a54754a6f6fdc73ceb4ab5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=57.9 K 2023-03-24 21:53:57,882 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56862 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:53:57,882 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:53:57,882 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,882 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/b551571ce1154987acf4c60349184c84, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=55.5 K 2023-03-24 21:53:57,882 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6474459208a54754a6f6fdc73ceb4ab5, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:57,883 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6aa062eaec4f4e9ea5231e6473ac8f69, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1679694834894 2023-03-24 21:53:57,883 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b551571ce1154987acf4c60349184c84, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:57,883 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 683a6ec756f3450599002aae34b4a6ae, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694835535 2023-03-24 21:53:57,883 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6373d6ee9b7343729209a406f8e14d6a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1679694834905 2023-03-24 21:53:57,884 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f3596ebe56d5474aa4ca834c322c11ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1679694836228 2023-03-24 21:53:57,884 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8a5b45d1d5fb466cb4ec8261642ce9d5, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694835535 2023-03-24 21:53:57,884 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c570bfc0c11d4355a958c02835bf48f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:53:57,886 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0c59c302cb6c46ea8f9aea147799867e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1679694836228 2023-03-24 21:53:57,886 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 15747cff1a914274a6b1ba022e181141, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:53:57,902 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#87 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:57,908 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#88 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:57,924 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/cbf45c65174e47589b0aba217def5d8f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/cbf45c65174e47589b0aba217def5d8f 2023-03-24 21:53:57,928 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/ebe31f567fd346fd9a62f6d681771d21 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ebe31f567fd346fd9a62f6d681771d21 2023-03-24 21:53:57,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into cbf45c65174e47589b0aba217def5d8f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:57,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:57,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=11, startTime=1679694837878; duration=0sec 2023-03-24 21:53:57,931 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:53:57,931 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:53:57,933 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56862 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:53:57,933 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:53:57,933 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,934 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ae0e33c3f4b7469da193f49db0b245fc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=55.5 K 2023-03-24 21:53:57,934 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ae0e33c3f4b7469da193f49db0b245fc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=694, earliestPutTs=1679694834266 2023-03-24 21:53:57,935 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 71d1635a677f44a8ae50c84b56986aee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1679694834905 2023-03-24 21:53:57,935 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting beabe816691a48c588a6461082910776, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694835535 2023-03-24 21:53:57,936 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a2e1723e86364e509bce7029f20d377d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=862, earliestPutTs=1679694836228 2023-03-24 21:53:57,936 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d0ce60b079324f3cac05ecd74afb3ecf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:53:57,936 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into ebe31f567fd346fd9a62f6d681771d21(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:57,936 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:57,936 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=11, startTime=1679694837877; duration=0sec 2023-03-24 21:53:57,937 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:57,947 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#89 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:53:57,956 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/31b40ca0aa7e42c7aed0ed95fff5654d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/31b40ca0aa7e42c7aed0ed95fff5654d 2023-03-24 21:53:57,961 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into 31b40ca0aa7e42c7aed0ed95fff5654d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:53:57,962 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:57,962 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=11, startTime=1679694837878; duration=0sec 2023-03-24 21:53:57,962 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:53:57,973 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:57,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:57,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:57,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:57,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:57,983 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:57,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:57,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:57,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:57,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:57,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:57,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:57,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:57,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:57,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:57,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:57,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:57,989 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:57,989 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:57,989 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:57,990 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:57,990 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-03-24 21:53:57,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:57,991 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:57,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:57,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:57,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:57,992 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:57,992 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:57,992 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:53:57,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694897998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:57,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:57,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694897999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694898053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,056 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694898054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694898101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,104 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694898102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694898304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694898306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,405 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9c2fe8c2ecce4111a93619468fb4871b 2023-03-24 21:53:58,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694898557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694898561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694898610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694898612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,821 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/55d9f5d75b8845c19de3b4217779925e 2023-03-24 21:53:58,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:58,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694898962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:58,965 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:53:59,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694899116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694899117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,243 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=912 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/0b46b390ae5a429ba1f6462ff131086a 2023-03-24 21:53:59,249 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9c2fe8c2ecce4111a93619468fb4871b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b 2023-03-24 21:53:59,256 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b, entries=150, sequenceid=912, filesize=11.9 K 2023-03-24 21:53:59,258 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/55d9f5d75b8845c19de3b4217779925e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e 2023-03-24 21:53:59,263 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e, entries=150, sequenceid=912, filesize=11.9 K 2023-03-24 21:53:59,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/0b46b390ae5a429ba1f6462ff131086a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a 2023-03-24 21:53:59,269 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a, entries=150, sequenceid=912, filesize=11.9 K 2023-03-24 21:53:59,270 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for a01d0abc8722ae28834e0262613a5783 in 1280ms, sequenceid=912, compaction requested=false 2023-03-24 21:53:59,270 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:59,270 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:59,270 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:59,270 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:59,270 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,271 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:59,271 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:59,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:59,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a14a9e4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:59,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:59,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:59,278 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:59,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:59,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:59,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:59,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:59,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:59,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:59,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:59,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:59,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:59,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:59,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:59,289 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:59,289 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:59,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:59,389 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:53:59,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:53:59,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:53:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:53:59,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:53:59,394 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:53:59,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:59,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:53:59,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:53:59,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,396 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:59,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:59,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:53:59,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:53:59,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:53:59,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:53:59,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:53:59,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:53:59,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:53:59,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:53:59,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:53:59,398 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:59,398 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:53:59,398 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 21:53:59,398 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:53:59,398 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:59,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:53:59,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:59,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:53:59,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:53:59,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:53:59,566 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694839574, max timestamp=1679694839589Num uniques 859; 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694839574, max timestamp=1679694839589Num uniques 859; 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694839574, max timestamp=1679694839589Num uniques 859; 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:53:59,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:53:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694899624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694899625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694899729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694899729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,815 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9a57f57cf629477384c59e9c76a4cc30 2023-03-24 21:53:59,854 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/7bbb4ae04aca4cc78690c9ca8cfa4d89 2023-03-24 21:53:59,900 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d71aba45e8154bd7b5747f1078974bbc 2023-03-24 21:53:59,909 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/9a57f57cf629477384c59e9c76a4cc30 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30 2023-03-24 21:53:59,917 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30, entries=150, sequenceid=920, filesize=11.9 K 2023-03-24 21:53:59,919 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/7bbb4ae04aca4cc78690c9ca8cfa4d89 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89 2023-03-24 21:53:59,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694899932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,935 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89, entries=150, sequenceid=920, filesize=11.9 K 2023-03-24 21:53:59,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:53:59,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694899934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,937 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/d71aba45e8154bd7b5747f1078974bbc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc 2023-03-24 21:53:59,950 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc, entries=150, sequenceid=920, filesize=11.9 K 2023-03-24 21:53:59,951 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for a01d0abc8722ae28834e0262613a5783 in 553ms, sequenceid=920, compaction requested=true 2023-03-24 21:53:59,951 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:53:59,951 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:53:59,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:53:59,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:53:59,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:53:59,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:53:59,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:53:59,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:53:59,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1289773e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:53:59,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:53:59,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,961 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:53:59,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:53:59,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:53:59,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:53:59,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:53:59,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:59,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:53:59,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:53:59,968 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:53:59,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:53:59,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:53:59,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:53:59,969 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:53:59,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:59,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:53:59,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:59,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:53:59,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:53:59,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:53:59,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:53:59,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:53:59,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:53:59,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:53:59,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:53:59,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:53:59,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:53:59,981 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:53:59,981 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:53:59,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:53:59,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:00,081 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:00,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:00,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:00,092 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:00,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:00,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:00,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,097 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:00,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:00,102 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:00,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:00,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:00,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:00,108 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:00,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:00,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:00,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:00,133 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=957 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/610e1b1905544f23878bc5087603b635 2023-03-24 21:54:00,137 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694900135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694900136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,151 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=957 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/fb2fe9f9d1b847979a3c309126c1870f 2023-03-24 21:54:00,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694900237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694900241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,242 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694900242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694900243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694900444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694900447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,570 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=957 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/b8dc325d7efe4701b4a424a7a427f7e1 2023-03-24 21:54:00,577 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/610e1b1905544f23878bc5087603b635 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635 2023-03-24 21:54:00,586 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635, entries=150, sequenceid=957, filesize=11.9 K 2023-03-24 21:54:00,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/fb2fe9f9d1b847979a3c309126c1870f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f 2023-03-24 21:54:00,602 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f, entries=150, sequenceid=957, filesize=11.9 K 2023-03-24 21:54:00,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/b8dc325d7efe4701b4a424a7a427f7e1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1 2023-03-24 21:54:00,620 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1, entries=150, sequenceid=957, filesize=11.9 K 2023-03-24 21:54:00,623 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for a01d0abc8722ae28834e0262613a5783 in 515ms, sequenceid=957, compaction requested=true 2023-03-24 21:54:00,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:00,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:00,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:00,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:00,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:00,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:00,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:00,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f976060[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:00,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:00,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,634 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,634 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:00,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:00,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:00,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:00,637 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:00,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:00,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:00,642 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:00,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:00,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:00,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:00,642 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:00,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:00,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:00,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,657 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:00,657 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:00,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:00,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:00,748 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:00,748 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:00,757 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:00,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:00,759 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:00,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:00,766 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:00,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:00,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:00,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,769 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:00,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,770 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:00,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:00,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:00,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:00,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:00,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:00,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:00,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:00,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:00,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:00,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:00,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:00,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:00,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:00,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:00,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360642c5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:00,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,776 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:00,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:00,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:00,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:00,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:00,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:00,780 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:00,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:00,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:00,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:00,780 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:00,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:00,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,787 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:00,787 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:00,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694840753, max timestamp=1679694840795Num uniques 859; 2023-03-24 21:54:00,796 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:00,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694840753, max timestamp=1679694840795Num uniques 859; 2023-03-24 21:54:00,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694840753, max timestamp=1679694840795Num uniques 859; 2023-03-24 21:54:00,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:00,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:00,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694900808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694900809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 367 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694900809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694900810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,887 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:00,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:00,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:00,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:00,891 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:00,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:00,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:00,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,893 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:00,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:00,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:00,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:00,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:00,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:00,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:00,895 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,895 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:00,895 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:00,895 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:00,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:00,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:00,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39529668[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:00,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:00,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:00,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:00,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:00,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:00,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:00,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:00,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:00,906 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:00,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:00,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:00,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:00,906 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:00,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:00,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:00,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:00,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:00,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:00,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:00,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:00,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:00,913 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:00,913 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:00,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694900911, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694900913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694900913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:00,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:00,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 369 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694900913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,013 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,018 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,023 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,023 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,023 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,023 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,024 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@603b26fb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,031 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,034 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,034 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,057 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,057 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:01,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694901115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,117 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694901115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 371 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694901115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694901116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,157 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,161 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,162 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,164 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,164 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,164 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,164 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,166 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e09086d4055f4448976ec6d5a2e20945 2023-03-24 21:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34fbdefc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,170 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,179 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,179 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,179 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,179 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,179 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,185 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:01,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,189 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,189 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,289 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,293 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,294 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30efa2cc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,304 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,309 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,309 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,316 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,316 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,416 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,420 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694901418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694901419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 373 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694901420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694901420, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,424 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,427 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1da3936f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,438 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,438 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,440 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,444 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,444 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,544 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,548 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,552 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,555 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,555 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,555 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,555 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,555 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,557 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@241672de[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,559 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,559 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,560 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,560 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,562 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,576 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,576 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,592 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/2015dd608021410cb1635a6e30fb94f4 2023-03-24 21:54:01,676 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,679 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,681 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,683 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,683 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,683 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,683 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef0449e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,693 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,693 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,695 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,695 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,695 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,700 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,700 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,701 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,701 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,701 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,800 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,804 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,808 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,808 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,808 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,808 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,808 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@344664e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,821 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,821 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,848 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,849 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694901924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,929 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694901925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 375 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694901927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:01,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694901928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,949 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:01,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:01,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:01,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:01,952 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:01,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:01,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:01,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,953 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,954 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,955 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:01,955 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:01,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:01,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:01,955 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:01,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:01,956 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:01,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:01,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:01,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fda5a2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:01,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:01,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,960 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:01,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:01,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:01,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:01,962 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:01,964 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:01,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:01,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:01,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:01,965 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:01,966 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:01,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:01,967 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,967 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:01,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:01,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:01,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:01,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:01,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:01,973 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:01,973 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:01,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:01,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,007 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=967 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/320f8aebdd984146aaaa7b17e9bd73bc 2023-03-24 21:54:02,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e09086d4055f4448976ec6d5a2e20945 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945 2023-03-24 21:54:02,017 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945, entries=100, sequenceid=967, filesize=9.5 K 2023-03-24 21:54:02,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/2015dd608021410cb1635a6e30fb94f4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4 2023-03-24 21:54:02,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4, entries=100, sequenceid=967, filesize=9.5 K 2023-03-24 21:54:02,024 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/320f8aebdd984146aaaa7b17e9bd73bc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc 2023-03-24 21:54:02,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc, entries=100, sequenceid=967, filesize=9.5 K 2023-03-24 21:54:02,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for a01d0abc8722ae28834e0262613a5783 in 1283ms, sequenceid=967, compaction requested=true 2023-03-24 21:54:02,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,031 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:02,032 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:02,033 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:02,033 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59474 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:02,033 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:02,034 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:54:02,034 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,034 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ebe31f567fd346fd9a62f6d681771d21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=58.1 K 2023-03-24 21:54:02,034 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ebe31f567fd346fd9a62f6d681771d21, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:54:02,035 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9c2fe8c2ecce4111a93619468fb4871b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1679694837444 2023-03-24 21:54:02,035 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9a57f57cf629477384c59e9c76a4cc30, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679694837993 2023-03-24 21:54:02,036 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 610e1b1905544f23878bc5087603b635, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1679694839620 2023-03-24 21:54:02,036 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59474 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:02,037 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:54:02,037 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,037 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/cbf45c65174e47589b0aba217def5d8f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=58.1 K 2023-03-24 21:54:02,037 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e09086d4055f4448976ec6d5a2e20945, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694840134 2023-03-24 21:54:02,038 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting cbf45c65174e47589b0aba217def5d8f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:54:02,038 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 55d9f5d75b8845c19de3b4217779925e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1679694837444 2023-03-24 21:54:02,039 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7bbb4ae04aca4cc78690c9ca8cfa4d89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679694837993 2023-03-24 21:54:02,040 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting fb2fe9f9d1b847979a3c309126c1870f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1679694839620 2023-03-24 21:54:02,041 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2015dd608021410cb1635a6e30fb94f4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694840134 2023-03-24 21:54:02,053 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#102 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:02,059 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#103 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:02,074 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,078 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,081 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,085 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/eaeb4966a7f948ea9a3f2e14abca19a7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/eaeb4966a7f948ea9a3f2e14abca19a7 2023-03-24 21:54:02,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,087 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:02,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:02,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:02,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:02,090 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into eaeb4966a7f948ea9a3f2e14abca19a7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:02,090 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,090 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=11, startTime=1679694842033; duration=0sec 2023-03-24 21:54:02,091 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:02,091 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:02,093 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59474 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:02,093 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:54:02,093 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,093 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/31b40ca0aa7e42c7aed0ed95fff5654d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=58.1 K 2023-03-24 21:54:02,093 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 31b40ca0aa7e42c7aed0ed95fff5654d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=872, earliestPutTs=1679694837368 2023-03-24 21:54:02,094 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0b46b390ae5a429ba1f6462ff131086a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=912, earliestPutTs=1679694837444 2023-03-24 21:54:02,094 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d71aba45e8154bd7b5747f1078974bbc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1679694837993 2023-03-24 21:54:02,095 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b8dc325d7efe4701b4a424a7a427f7e1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=957, earliestPutTs=1679694839620 2023-03-24 21:54:02,095 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 320f8aebdd984146aaaa7b17e9bd73bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694840134 2023-03-24 21:54:02,106 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1001 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/59e20c5528bf4702bf1cc262251d0751 2023-03-24 21:54:02,108 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:02,133 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1001 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6dadc8f9610e4f54a30782f75488eccb 2023-03-24 21:54:02,146 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1001 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/ce3808f928e049fdbbf311d06a08b96e 2023-03-24 21:54:02,151 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/59e20c5528bf4702bf1cc262251d0751 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751 2023-03-24 21:54:02,156 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751, entries=150, sequenceid=1001, filesize=11.9 K 2023-03-24 21:54:02,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6dadc8f9610e4f54a30782f75488eccb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb 2023-03-24 21:54:02,165 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb, entries=150, sequenceid=1001, filesize=11.9 K 2023-03-24 21:54:02,165 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/ce3808f928e049fdbbf311d06a08b96e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e 2023-03-24 21:54:02,170 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e, entries=150, sequenceid=1001, filesize=11.9 K 2023-03-24 21:54:02,171 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=0 B/0 for a01d0abc8722ae28834e0262613a5783 in 84ms, sequenceid=1001, compaction requested=false 2023-03-24 21:54:02,171 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,171 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,171 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,174 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,174 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,174 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,176 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15eaa7a1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,177 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,177 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,179 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,181 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,182 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,185 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,185 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,185 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,185 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,189 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,190 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,190 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,290 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,293 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,295 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,299 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5972d536[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,304 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,308 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,308 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,312 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,427 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,438 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79a4ae40[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,450 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,450 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,458 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,458 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,458 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,480 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/c1961327cce34a37aafc2d2a7cf5a591 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c1961327cce34a37aafc2d2a7cf5a591 2023-03-24 21:54:02,487 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into c1961327cce34a37aafc2d2a7cf5a591(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:02,487 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,487 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=11, startTime=1679694842031; duration=0sec 2023-03-24 21:54:02,487 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:02,527 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/f22a8ba59b484fcfa9526f0b7ee8f1d9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f22a8ba59b484fcfa9526f0b7ee8f1d9 2023-03-24 21:54:02,533 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into f22a8ba59b484fcfa9526f0b7ee8f1d9(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:02,533 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,533 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=11, startTime=1679694842033; duration=0sec 2023-03-24 21:54:02,533 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:02,558 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,562 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,564 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,565 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,565 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,565 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,565 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,566 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,566 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,566 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,566 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,570 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@169bc463[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,577 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,578 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,584 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,584 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,688 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,692 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,692 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,692 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,692 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,695 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568caebe[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,703 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,703 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,707 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,710 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,710 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,810 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,817 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,820 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,822 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,822 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:02,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,827 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,827 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,831 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19181a3c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,832 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,839 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,839 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,851 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,851 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:02,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:02,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:02,937 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:02,937 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:02,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:02,938 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:02,938 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:02,949 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/2353bfce2acc4011ad90cb47dead3174 2023-03-24 21:54:02,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:02,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:02,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:02,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:02,961 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:02,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:02,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:02,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,965 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/1c36116b8742499ebd5e110109033189 2023-03-24 21:54:02,971 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:02,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:02,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:02,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:02,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:02,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:02,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:02,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:02,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:02,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:02,975 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,975 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:02,975 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:02,975 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:02,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:02,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:02,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:02,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:02,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694842953, max timestamp=1679694842962Num uniques 859; 2023-03-24 21:54:02,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 21:54:02,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694842953, max timestamp=1679694842962Num uniques 859; 2023-03-24 21:54:02,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:02,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694842953, max timestamp=1679694842962Num uniques 859; 2023-03-24 21:54:02,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:02,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1020 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/413ff6055d04439d97231501e18d5d2e 2023-03-24 21:54:02,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:02,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@459261f1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:02,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:02,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:02,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:02,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:02,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/2353bfce2acc4011ad90cb47dead3174 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174 2023-03-24 21:54:02,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:02,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:02,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:02,986 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:02,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:02,989 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:02,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:02,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:02,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:02,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:02,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174, entries=150, sequenceid=1020, filesize=11.9 K 2023-03-24 21:54:02,990 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/1c36116b8742499ebd5e110109033189 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189 2023-03-24 21:54:02,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:02,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:02,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:02,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:02,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189, entries=150, sequenceid=1020, filesize=11.9 K 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:02,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:02,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:02,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:02,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:02,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:02,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/413ff6055d04439d97231501e18d5d2e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e 2023-03-24 21:54:02,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:02,998 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:02,998 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:02,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:02,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,012 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e, entries=150, sequenceid=1020, filesize=11.9 K 2023-03-24 21:54:03,013 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=46.57 KB/47685 for a01d0abc8722ae28834e0262613a5783 in 76ms, sequenceid=1020, compaction requested=true 2023-03-24 21:54:03,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:03,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:03,013 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:03,013 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:03,013 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:03,014 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:03,015 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:03,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:03,015 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/B is initiating minor compaction (all files) 2023-03-24 21:54:03,015 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:03,015 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/B in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,015 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/A is initiating minor compaction (all files) 2023-03-24 21:54:03,015 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/eaeb4966a7f948ea9a3f2e14abca19a7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.8 K 2023-03-24 21:54:03,016 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=53.28 KB heapSize=145.01 KB 2023-03-24 21:54:03,015 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/A in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,016 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting eaeb4966a7f948ea9a3f2e14abca19a7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694839620 2023-03-24 21:54:03,016 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c1961327cce34a37aafc2d2a7cf5a591, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.8 K 2023-03-24 21:54:03,016 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6dadc8f9610e4f54a30782f75488eccb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1001, earliestPutTs=1679694840804 2023-03-24 21:54:03,017 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c1961327cce34a37aafc2d2a7cf5a591, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694839620 2023-03-24 21:54:03,017 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1c36116b8742499ebd5e110109033189, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1679694842935 2023-03-24 21:54:03,018 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 59e20c5528bf4702bf1cc262251d0751, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1001, earliestPutTs=1679694840804 2023-03-24 21:54:03,018 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2353bfce2acc4011ad90cb47dead3174, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1679694842935 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:03,019 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:03,027 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#A#compaction#111 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:03,040 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#B#compaction#112 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:03,048 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694843032, max timestamp=1679694843045Num uniques 859; 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694843032, max timestamp=1679694843045Num uniques 859; 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679694843032, max timestamp=1679694843045Num uniques 859; 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694903039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,048 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-24 21:54:03,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694903063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694903063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694903064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,069 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/059420cb40e946c3b9352ca547292720 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/059420cb40e946c3b9352ca547292720 2023-03-24 21:54:03,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694903066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,074 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/A of a01d0abc8722ae28834e0262613a5783 into 059420cb40e946c3b9352ca547292720(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:03,074 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:03,074 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/A, priority=13, startTime=1679694843013; duration=0sec 2023-03-24 21:54:03,075 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:03,075 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:03,076 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:03,076 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): a01d0abc8722ae28834e0262613a5783/C is initiating minor compaction (all files) 2023-03-24 21:54:03,076 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of a01d0abc8722ae28834e0262613a5783/C in TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,076 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f22a8ba59b484fcfa9526f0b7ee8f1d9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp, totalSize=36.8 K 2023-03-24 21:54:03,077 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f22a8ba59b484fcfa9526f0b7ee8f1d9, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=967, earliestPutTs=1679694839620 2023-03-24 21:54:03,077 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ce3808f928e049fdbbf311d06a08b96e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1001, earliestPutTs=1679694840804 2023-03-24 21:54:03,078 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 413ff6055d04439d97231501e18d5d2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1020, earliestPutTs=1679694842935 2023-03-24 21:54:03,084 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): a01d0abc8722ae28834e0262613a5783#C#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:03,098 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,101 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,104 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,104 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,104 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,104 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,104 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,105 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,105 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,106 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d620dfc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,110 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,115 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,115 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,118 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,118 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,118 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,123 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,124 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694903149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694903167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694903168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694903168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694903171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,224 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,227 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,234 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ea8abb0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,257 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,257 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,260 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,265 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,265 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694903356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,365 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,368 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,371 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,373 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,373 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,373 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,373 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,373 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,373 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,373 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,373 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,373 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 394 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694903373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,375 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694903373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694903374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694903374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b89fa3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,382 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,383 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,386 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,386 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,388 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,395 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,395 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,395 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,454 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/210577001f0844c7a93b74518fb5fc9f 2023-03-24 21:54:03,474 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/85bd482d140d4c0e9748ac11099ac581 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/85bd482d140d4c0e9748ac11099ac581 2023-03-24 21:54:03,482 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/B of a01d0abc8722ae28834e0262613a5783 into 85bd482d140d4c0e9748ac11099ac581(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:03,482 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:03,482 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/B, priority=13, startTime=1679694843013; duration=0sec 2023-03-24 21:54:03,483 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:03,509 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,516 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,519 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/e8a6bf6bcf0a49e0ad6eff64a43da945 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e8a6bf6bcf0a49e0ad6eff64a43da945 2023-03-24 21:54:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,523 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,523 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,523 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,526 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in a01d0abc8722ae28834e0262613a5783/C of a01d0abc8722ae28834e0262613a5783 into e8a6bf6bcf0a49e0ad6eff64a43da945(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:03,526 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:03,526 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783., storeName=a01d0abc8722ae28834e0262613a5783/C, priority=13, startTime=1679694843014; duration=0sec 2023-03-24 21:54:03,526 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:03,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f3f74c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,530 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,530 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,530 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,535 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,535 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,541 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,541 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,653 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,653 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,653 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ce31163[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,661 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694903660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,662 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,662 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,662 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,666 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,666 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,669 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,670 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,674 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,675 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,675 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694903677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694903677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694903678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:03,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694903679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,775 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,778 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,781 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,781 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,782 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,782 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,782 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,782 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,782 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576dca64[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,787 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,787 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,789 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,789 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,790 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,792 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,792 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,801 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,801 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/3a351706d6e74ed999ae4221e5031350 2023-03-24 21:54:03,901 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:03,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:03,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:03,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:03,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:03,904 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:03,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:03,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:03,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:03,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:03,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,908 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:03,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:03,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:03,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:03,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:03,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:03,909 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,909 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. started... 2023-03-24 21:54:03,909 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. as already flushing 2023-03-24 21:54:03,909 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:03,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:03,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,911 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:03,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:03,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@740b02ad[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:03,913 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:03,913 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,914 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:03,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:03,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:03,915 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,915 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:03,916 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:03,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:03,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:03,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:03,919 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:03,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:03,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:03,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:03,919 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:03,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:03,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:03,924 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:03,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:03,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:03,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:03,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:03,927 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:03,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:03,928 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:03,928 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:03,928 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:03,928 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:03,928 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,928 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:03,928 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:03,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:03,954 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5cc86371 to 127.0.0.1:57724 2023-03-24 21:54:03,954 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:03,956 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5eb3f68d to 127.0.0.1:57724 2023-03-24 21:54:03,956 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:03,963 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3625ffd0 to 127.0.0.1:57724 2023-03-24 21:54:03,963 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b52d1b4 to 127.0.0.1:57724 2023-03-24 21:54:03,963 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:03,963 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:04,028 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:04,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:04,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:04,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38300 deadline: 1679694904167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:04,184 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:04,184 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:04,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38322 deadline: 1679694904184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:04,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38276 deadline: 1679694904184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:04,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:04,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38312 deadline: 1679694904185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:04,186 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:04,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:38324 deadline: 1679694904186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:04,287 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=19.73 KB at sequenceid=1046 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/81cc713bd23343d0aeed11132142de6f 2023-03-24 21:54:04,292 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/210577001f0844c7a93b74518fb5fc9f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/210577001f0844c7a93b74518fb5fc9f 2023-03-24 21:54:04,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/210577001f0844c7a93b74518fb5fc9f, entries=150, sequenceid=1046, filesize=11.9 K 2023-03-24 21:54:04,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/3a351706d6e74ed999ae4221e5031350 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3a351706d6e74ed999ae4221e5031350 2023-03-24 21:54:04,300 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3a351706d6e74ed999ae4221e5031350, entries=150, sequenceid=1046, filesize=11.9 K 2023-03-24 21:54:04,301 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/81cc713bd23343d0aeed11132142de6f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/81cc713bd23343d0aeed11132142de6f 2023-03-24 21:54:04,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/81cc713bd23343d0aeed11132142de6f, entries=150, sequenceid=1046, filesize=11.9 K 2023-03-24 21:54:04,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~59.99 KB/61425, heapSize ~164.88 KB/168840, currentSize=134.18 KB/137400 for a01d0abc8722ae28834e0262613a5783 in 1291ms, sequenceid=1046, compaction requested=false 2023-03-24 21:54:04,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:05,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:05,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=140.89 KB heapSize=386.16 KB 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:05,174 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x258416be to 127.0.0.1:57724 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:05,174 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:05,174 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:05,192 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ec02d2b to 127.0.0.1:57724 2023-03-24 21:54:05,192 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09417ce8 to 127.0.0.1:57724 2023-03-24 21:54:05,192 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x634d1248 to 127.0.0.1:57724 2023-03-24 21:54:05,192 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,192 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,192 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,193 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30a6dfbc to 127.0.0.1:57724 2023-03-24 21:54:05,194 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 268 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 168 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 158 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 221 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 146 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4243 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4207 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1847 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5532 rows 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1874 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5621 rows 2023-03-24 21:54:05,194 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:54:05,194 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4507c0c2 to 127.0.0.1:57724 2023-03-24 21:54:05,195 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:05,200 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:54:05,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:54:05,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:05,218 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694845217"}]},"ts":"1679694845217"} 2023-03-24 21:54:05,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:05,219 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:54:05,301 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:54:05,304 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, UNASSIGN}] 2023-03-24 21:54:05,305 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, UNASSIGN 2023-03-24 21:54:05,306 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a01d0abc8722ae28834e0262613a5783, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:05,306 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694845306"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694845306"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694845306"}]},"ts":"1679694845306"} 2023-03-24 21:54:05,308 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:05,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:05,466 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:05,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing a01d0abc8722ae28834e0262613a5783, disabling compactions & flushes 2023-03-24 21:54:05,467 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:05,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:05,582 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e85901cf92d94352a69ec1180f3bb754 2023-03-24 21:54:05,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:05,993 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/8f3fbf9bda0146cda5c747f05e2d89d6 2023-03-24 21:54:06,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:06,404 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=1083 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/6c896953c50d4ff9b5789bb569a5e09a 2023-03-24 21:54:06,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/e85901cf92d94352a69ec1180f3bb754 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e85901cf92d94352a69ec1180f3bb754 2023-03-24 21:54:06,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e85901cf92d94352a69ec1180f3bb754, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-24 21:54:06,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/8f3fbf9bda0146cda5c747f05e2d89d6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8f3fbf9bda0146cda5c747f05e2d89d6 2023-03-24 21:54:06,419 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8f3fbf9bda0146cda5c747f05e2d89d6, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-24 21:54:06,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/6c896953c50d4ff9b5789bb569a5e09a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/6c896953c50d4ff9b5789bb569a5e09a 2023-03-24 21:54:06,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/6c896953c50d4ff9b5789bb569a5e09a, entries=150, sequenceid=1083, filesize=11.9 K 2023-03-24 21:54:06,426 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~386.11 KB/395376, currentSize=26.84 KB/27480 for a01d0abc8722ae28834e0262613a5783 in 1253ms, sequenceid=1083, compaction requested=true 2023-03-24 21:54:06,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:06,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:06,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:06,426 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. because compaction request was cancelled 2023-03-24 21:54:06,426 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. because compaction request was cancelled 2023-03-24 21:54:06,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:06,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:06,426 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:06,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. after waiting 0 ms 2023-03-24 21:54:06,426 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:06,426 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. because compaction request was cancelled 2023-03-24 21:54:06,426 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing a01d0abc8722ae28834e0262613a5783 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=A 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=B 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK a01d0abc8722ae28834e0262613a5783, store=C 2023-03-24 21:54:06,427 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:06,834 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1090 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/97861f031e2545e2a7ef7f269e3267db 2023-03-24 21:54:07,246 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1090 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6d9c85a616f14be580f25802d166f6b9 2023-03-24 21:54:07,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:07,656 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1090 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/e27461766c70463389f15f30df7dc2c4 2023-03-24 21:54:07,661 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/A/97861f031e2545e2a7ef7f269e3267db as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/97861f031e2545e2a7ef7f269e3267db 2023-03-24 21:54:07,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/97861f031e2545e2a7ef7f269e3267db, entries=150, sequenceid=1090, filesize=11.9 K 2023-03-24 21:54:07,667 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/B/6d9c85a616f14be580f25802d166f6b9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6d9c85a616f14be580f25802d166f6b9 2023-03-24 21:54:07,672 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6d9c85a616f14be580f25802d166f6b9, entries=150, sequenceid=1090, filesize=11.9 K 2023-03-24 21:54:07,672 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/.tmp/C/e27461766c70463389f15f30df7dc2c4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e27461766c70463389f15f30df7dc2c4 2023-03-24 21:54:07,678 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e27461766c70463389f15f30df7dc2c4, entries=150, sequenceid=1090, filesize=11.9 K 2023-03-24 21:54:07,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for a01d0abc8722ae28834e0262613a5783 in 1253ms, sequenceid=1090, compaction requested=true 2023-03-24 21:54:07,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/272d8fca6af4423b835bf4e87be54a7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/970aaa552d4d4c84a833f7e18dc6171e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e043a22d253c40dc8a69d945ac658f54, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/d5a220eb153a494ea8139d39e3e1a67c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/77c43b8e793748d3862e6e2ab66648b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5b8cb6e2165f4e9bbd5d15dc1c519c25, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6474459208a54754a6f6fdc73ceb4ab5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ebe31f567fd346fd9a62f6d681771d21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c1961327cce34a37aafc2d2a7cf5a591, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174] to archive 2023-03-24 21:54:07,703 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:07,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6d4fd6b7dd9b484fb901709b0f5186d6 2023-03-24 21:54:07,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5546db00b6c340458880157e38a520b5 2023-03-24 21:54:07,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/cbc563f1bf4f49f8b14963c14d3f4940 2023-03-24 21:54:07,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/272d8fca6af4423b835bf4e87be54a7b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/272d8fca6af4423b835bf4e87be54a7b 2023-03-24 21:54:07,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/7d438bf269ec4f8eb689168853ab0ad8 2023-03-24 21:54:07,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ff4db27172024adb9dcf8563746a9ec4 2023-03-24 21:54:07,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/970aaa552d4d4c84a833f7e18dc6171e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/970aaa552d4d4c84a833f7e18dc6171e 2023-03-24 21:54:07,717 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e43b8510d842428bb43c5edd92ba206d 2023-03-24 21:54:07,718 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ab78455cd89f496ca77069baec7c191a 2023-03-24 21:54:07,720 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/87dfc7d43ba94e7e8ab8f6f2e6cfa60c 2023-03-24 21:54:07,721 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/14b207781cc34dc988403823f720bb96 2023-03-24 21:54:07,722 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e043a22d253c40dc8a69d945ac658f54 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e043a22d253c40dc8a69d945ac658f54 2023-03-24 21:54:07,723 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/dc4eced99234410babac91cd2267f3d0 2023-03-24 21:54:07,724 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/af9dfd7ddbc644f09ecbff07e99c1512 2023-03-24 21:54:07,725 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/d5a220eb153a494ea8139d39e3e1a67c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/d5a220eb153a494ea8139d39e3e1a67c 2023-03-24 21:54:07,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c62de5488327455bbb52195fe9f67bcd 2023-03-24 21:54:07,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/77c43b8e793748d3862e6e2ab66648b0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/77c43b8e793748d3862e6e2ab66648b0 2023-03-24 21:54:07,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/706fb5c45b16474185326c7384940321 2023-03-24 21:54:07,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/3e83ccef651a4b61b09ccbf48e977f04 2023-03-24 21:54:07,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5b8cb6e2165f4e9bbd5d15dc1c519c25 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/5b8cb6e2165f4e9bbd5d15dc1c519c25 2023-03-24 21:54:07,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/062eb8c3bb8746a4ab5cd978a1986d3c 2023-03-24 21:54:07,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9e604678655646ec9e2c76d131a10e8b 2023-03-24 21:54:07,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9b53a4d617ca4a1090017174e0db3b4f 2023-03-24 21:54:07,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6474459208a54754a6f6fdc73ceb4ab5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6474459208a54754a6f6fdc73ceb4ab5 2023-03-24 21:54:07,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/1927f8457d0c49e0b67293b93f210787 2023-03-24 21:54:07,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/6aa062eaec4f4e9ea5231e6473ac8f69 2023-03-24 21:54:07,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/683a6ec756f3450599002aae34b4a6ae 2023-03-24 21:54:07,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/f3596ebe56d5474aa4ca834c322c11ca 2023-03-24 21:54:07,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ebe31f567fd346fd9a62f6d681771d21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/ebe31f567fd346fd9a62f6d681771d21 2023-03-24 21:54:07,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c570bfc0c11d4355a958c02835bf48f0 2023-03-24 21:54:07,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9c2fe8c2ecce4111a93619468fb4871b 2023-03-24 21:54:07,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/9a57f57cf629477384c59e9c76a4cc30 2023-03-24 21:54:07,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/610e1b1905544f23878bc5087603b635 2023-03-24 21:54:07,745 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c1961327cce34a37aafc2d2a7cf5a591 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/c1961327cce34a37aafc2d2a7cf5a591 2023-03-24 21:54:07,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e09086d4055f4448976ec6d5a2e20945 2023-03-24 21:54:07,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/59e20c5528bf4702bf1cc262251d0751 2023-03-24 21:54:07,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/2353bfce2acc4011ad90cb47dead3174 2023-03-24 21:54:07,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/662f5e7ddc6349639835bb3a9ca1aee4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6ca83a8f0e3447fe8f56caa977cd944f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3bd86db0e4a54b5692c917a70737f6b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/4dffb72c7eb141119c40dc1a6fac7b21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c655cd375170400e9d39e1bae8eab599, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/924a890a41c7428d9c17b13846982e5c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/b551571ce1154987acf4c60349184c84, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/cbf45c65174e47589b0aba217def5d8f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/eaeb4966a7f948ea9a3f2e14abca19a7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189] to archive 2023-03-24 21:54:07,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:07,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3c867a3e2a1645158a3e1e7780007ba4 2023-03-24 21:54:07,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/75aa24a4edd84cfb9cb597c9f6229edd 2023-03-24 21:54:07,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/54c4359e58f943719cdf2e6085df0495 2023-03-24 21:54:07,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/662f5e7ddc6349639835bb3a9ca1aee4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/662f5e7ddc6349639835bb3a9ca1aee4 2023-03-24 21:54:07,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/d50519d6bfba4206a3f5d71a0a0cdfa5 2023-03-24 21:54:07,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/72b5470bb0844731b22ab041c8e4ce0d 2023-03-24 21:54:07,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6ca83a8f0e3447fe8f56caa977cd944f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6ca83a8f0e3447fe8f56caa977cd944f 2023-03-24 21:54:07,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6c375d745e3d49049b3c12a844f50001 2023-03-24 21:54:07,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1ab097b7cb846e8b77c065aadea862e 2023-03-24 21:54:07,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/919451269ddb4283ae37bbf95978b9f9 2023-03-24 21:54:07,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3bd86db0e4a54b5692c917a70737f6b1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3bd86db0e4a54b5692c917a70737f6b1 2023-03-24 21:54:07,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c1a3b3ab60f0424692c0713c4fa9fd9e 2023-03-24 21:54:07,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/59fbfad3bb99413db7fd989307b0ddbe 2023-03-24 21:54:07,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/4dffb72c7eb141119c40dc1a6fac7b21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/4dffb72c7eb141119c40dc1a6fac7b21 2023-03-24 21:54:07,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55baa1eba81e4b0ead59bec9bbd0a898 2023-03-24 21:54:07,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0bc391bb3fdf46e2af3fc22904ef113a 2023-03-24 21:54:07,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c655cd375170400e9d39e1bae8eab599 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c655cd375170400e9d39e1bae8eab599 2023-03-24 21:54:07,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/9f54bee61e5f40df81983b5e6ec8a0aa 2023-03-24 21:54:07,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/c9f56563113b4c44a3c2aff6596710bd 2023-03-24 21:54:07,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/924a890a41c7428d9c17b13846982e5c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/924a890a41c7428d9c17b13846982e5c 2023-03-24 21:54:07,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/aa70253a2c9c41b880f9ebf2c2affcb0 2023-03-24 21:54:07,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/5a9b30df673f494e8007c0c67375ac76 2023-03-24 21:54:07,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2cefed0c48884b73aac4611b429108d8 2023-03-24 21:54:07,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/b551571ce1154987acf4c60349184c84 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/b551571ce1154987acf4c60349184c84 2023-03-24 21:54:07,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/65314cb743e64ac4a71efa86bd6fe9fe 2023-03-24 21:54:07,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6373d6ee9b7343729209a406f8e14d6a 2023-03-24 21:54:07,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8a5b45d1d5fb466cb4ec8261642ce9d5 2023-03-24 21:54:07,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/0c59c302cb6c46ea8f9aea147799867e 2023-03-24 21:54:07,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/cbf45c65174e47589b0aba217def5d8f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/cbf45c65174e47589b0aba217def5d8f 2023-03-24 21:54:07,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/15747cff1a914274a6b1ba022e181141 2023-03-24 21:54:07,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/55d9f5d75b8845c19de3b4217779925e 2023-03-24 21:54:07,808 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/7bbb4ae04aca4cc78690c9ca8cfa4d89 2023-03-24 21:54:07,809 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/fb2fe9f9d1b847979a3c309126c1870f 2023-03-24 21:54:07,810 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/eaeb4966a7f948ea9a3f2e14abca19a7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/eaeb4966a7f948ea9a3f2e14abca19a7 2023-03-24 21:54:07,811 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/2015dd608021410cb1635a6e30fb94f4 2023-03-24 21:54:07,813 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6dadc8f9610e4f54a30782f75488eccb 2023-03-24 21:54:07,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/1c36116b8742499ebd5e110109033189 2023-03-24 21:54:07,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/915cfffb745b4d70bb9e33cde1383b6e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5c4dafcf0524417491db0e36ea2dd715, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4e482a349a744a6781b734c5dd9ace21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/27574796d44e437c8d810b70693c18b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0c9663e55a1c4a98b6b65d7e91b99de0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a88e04e06dde4c2998f60675e3d84ff9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ae0e33c3f4b7469da193f49db0b245fc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/31b40ca0aa7e42c7aed0ed95fff5654d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f22a8ba59b484fcfa9526f0b7ee8f1d9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e] to archive 2023-03-24 21:54:07,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:07,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4ae9a01f7e3342c0a1597b877d86cc55 2023-03-24 21:54:07,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/18282588fc734896b707e2e9eae6b6d7 2023-03-24 21:54:07,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e89fa5cd38a740b693f639fbdc555a92 2023-03-24 21:54:07,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/915cfffb745b4d70bb9e33cde1383b6e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/915cfffb745b4d70bb9e33cde1383b6e 2023-03-24 21:54:07,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d205b7e24a8149f89e3e40cf6ca78cb2 2023-03-24 21:54:07,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/48df6a483ef04b4a9cc83e6927b8ae57 2023-03-24 21:54:07,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5c4dafcf0524417491db0e36ea2dd715 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5c4dafcf0524417491db0e36ea2dd715 2023-03-24 21:54:07,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/7e6a6642b0994b2fa4c44cc4a8af601e 2023-03-24 21:54:07,829 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f829e224397a4909abd64e3a5f7c2e98 2023-03-24 21:54:07,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/bf0d1705e9854a669c8fa6ea8b8a4c17 2023-03-24 21:54:07,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4e482a349a744a6781b734c5dd9ace21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4e482a349a744a6781b734c5dd9ace21 2023-03-24 21:54:07,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/24c97ba657424bb4b47bb606ec783c69 2023-03-24 21:54:07,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/5b66ba1743ca40489eeacc01b7dfee21 2023-03-24 21:54:07,834 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/27574796d44e437c8d810b70693c18b1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/27574796d44e437c8d810b70693c18b1 2023-03-24 21:54:07,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/1db5db2f9c5845068d04ce143d6ed7c7 2023-03-24 21:54:07,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/62e8f004855b4e759002dbb8a93945cc 2023-03-24 21:54:07,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0c9663e55a1c4a98b6b65d7e91b99de0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0c9663e55a1c4a98b6b65d7e91b99de0 2023-03-24 21:54:07,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3ddaf1ec735b41a79a549021b0bdc323 2023-03-24 21:54:07,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/3c956d25e71246d6bb3ce8a5bdae4124 2023-03-24 21:54:07,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a88e04e06dde4c2998f60675e3d84ff9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a88e04e06dde4c2998f60675e3d84ff9 2023-03-24 21:54:07,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/50c2f3447be147cc81f7253a8026f88d 2023-03-24 21:54:07,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0593f607e5ff4452a0e65ee4a1c23946 2023-03-24 21:54:07,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/775d93e536784a6d910994383f60cfeb 2023-03-24 21:54:07,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ae0e33c3f4b7469da193f49db0b245fc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ae0e33c3f4b7469da193f49db0b245fc 2023-03-24 21:54:07,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/4f72f3f3d053448ab33856c10a4eb68c 2023-03-24 21:54:07,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/71d1635a677f44a8ae50c84b56986aee 2023-03-24 21:54:07,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/beabe816691a48c588a6461082910776 2023-03-24 21:54:07,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/a2e1723e86364e509bce7029f20d377d 2023-03-24 21:54:07,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/31b40ca0aa7e42c7aed0ed95fff5654d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/31b40ca0aa7e42c7aed0ed95fff5654d 2023-03-24 21:54:07,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d0ce60b079324f3cac05ecd74afb3ecf 2023-03-24 21:54:07,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/0b46b390ae5a429ba1f6462ff131086a 2023-03-24 21:54:07,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/d71aba45e8154bd7b5747f1078974bbc 2023-03-24 21:54:07,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/b8dc325d7efe4701b4a424a7a427f7e1 2023-03-24 21:54:07,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f22a8ba59b484fcfa9526f0b7ee8f1d9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/f22a8ba59b484fcfa9526f0b7ee8f1d9 2023-03-24 21:54:07,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/320f8aebdd984146aaaa7b17e9bd73bc 2023-03-24 21:54:07,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/ce3808f928e049fdbbf311d06a08b96e 2023-03-24 21:54:07,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/413ff6055d04439d97231501e18d5d2e 2023-03-24 21:54:07,862 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/recovered.edits/1093.seqid, newMaxSeqId=1093, maxSeqId=1 2023-03-24 21:54:07,864 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783. 2023-03-24 21:54:07,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for a01d0abc8722ae28834e0262613a5783: 2023-03-24 21:54:07,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:07,866 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=a01d0abc8722ae28834e0262613a5783, regionState=CLOSED 2023-03-24 21:54:07,867 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694847866"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694847866"}]},"ts":"1679694847866"} 2023-03-24 21:54:07,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-24 21:54:07,869 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure a01d0abc8722ae28834e0262613a5783, server=jenkins-hbase9.apache.org,45673,1679694819052 in 2.5600 sec 2023-03-24 21:54:07,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-24 21:54:07,870 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=a01d0abc8722ae28834e0262613a5783, UNASSIGN in 2.5660 sec 2023-03-24 21:54:07,871 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694847871"}]},"ts":"1679694847871"} 2023-03-24 21:54:07,872 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:54:07,874 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:54:07,876 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6660 sec 2023-03-24 21:54:09,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-24 21:54:09,324 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-24 21:54:09,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:54:09,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,337 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,338 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 21:54:09,343 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:09,345 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/recovered.edits] 2023-03-24 21:54:09,348 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/059420cb40e946c3b9352ca547292720 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/059420cb40e946c3b9352ca547292720 2023-03-24 21:54:09,350 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/210577001f0844c7a93b74518fb5fc9f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/210577001f0844c7a93b74518fb5fc9f 2023-03-24 21:54:09,352 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/97861f031e2545e2a7ef7f269e3267db to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/97861f031e2545e2a7ef7f269e3267db 2023-03-24 21:54:09,353 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e85901cf92d94352a69ec1180f3bb754 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/A/e85901cf92d94352a69ec1180f3bb754 2023-03-24 21:54:09,356 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3a351706d6e74ed999ae4221e5031350 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/3a351706d6e74ed999ae4221e5031350 2023-03-24 21:54:09,357 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6d9c85a616f14be580f25802d166f6b9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/6d9c85a616f14be580f25802d166f6b9 2023-03-24 21:54:09,358 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/85bd482d140d4c0e9748ac11099ac581 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/85bd482d140d4c0e9748ac11099ac581 2023-03-24 21:54:09,359 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8f3fbf9bda0146cda5c747f05e2d89d6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/B/8f3fbf9bda0146cda5c747f05e2d89d6 2023-03-24 21:54:09,362 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/6c896953c50d4ff9b5789bb569a5e09a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/6c896953c50d4ff9b5789bb569a5e09a 2023-03-24 21:54:09,363 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/81cc713bd23343d0aeed11132142de6f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/81cc713bd23343d0aeed11132142de6f 2023-03-24 21:54:09,364 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e27461766c70463389f15f30df7dc2c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e27461766c70463389f15f30df7dc2c4 2023-03-24 21:54:09,365 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e8a6bf6bcf0a49e0ad6eff64a43da945 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/C/e8a6bf6bcf0a49e0ad6eff64a43da945 2023-03-24 21:54:09,368 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/recovered.edits/1093.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783/recovered.edits/1093.seqid 2023-03-24 21:54:09,369 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/a01d0abc8722ae28834e0262613a5783 2023-03-24 21:54:09,369 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:54:09,372 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,395 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:54:09,439 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:54:09,440 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,441 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:54:09,441 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694849441"}]},"ts":"9223372036854775807"} 2023-03-24 21:54:09,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 21:54:09,443 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:54:09,443 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => a01d0abc8722ae28834e0262613a5783, NAME => 'TestAcidGuarantees,,1679694823190.a01d0abc8722ae28834e0262613a5783.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:54:09,443 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:54:09,444 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694849443"}]},"ts":"9223372036854775807"} 2023-03-24 21:54:09,445 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:54:09,447 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,449 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 117 msec 2023-03-24 21:54:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-24 21:54:09,642 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-24 21:54:09,659 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase9:45673-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x770a4cd5-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x770a4cd5-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x770a4cd5-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1314623677_15 at /127.0.0.1:57862 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1314623677_15 at /127.0.0.1:57788 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x770a4cd5-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=563 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=370 (was 291) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 168), AvailableMemoryMB=8392 (was 8966) 2023-03-24 21:54:09,670 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=563, MaxFileDescriptor=60000, SystemLoadAverage=370, ProcessCount=168, AvailableMemoryMB=8392 2023-03-24 21:54:09,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:54:09,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:54:09,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:09,676 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:54:09,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-24 21:54:09,677 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:54:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:09,678 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:09,679 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c empty. 2023-03-24 21:54:09,679 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:09,679 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:54:09,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:09,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:10,088 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:54:10,090 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 733bfd60f65777910098447a7a228a5c, NAME => 'TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:54:10,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 733bfd60f65777910098447a7a228a5c, disabling compactions & flushes 2023-03-24 21:54:10,496 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. after waiting 0 ms 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,496 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,496 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:10,499 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:54:10,500 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694850499"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694850499"}]},"ts":"1679694850499"} 2023-03-24 21:54:10,501 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:54:10,502 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:54:10,502 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694850502"}]},"ts":"1679694850502"} 2023-03-24 21:54:10,503 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:54:10,511 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, ASSIGN}] 2023-03-24 21:54:10,513 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, ASSIGN 2023-03-24 21:54:10,514 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:54:10,665 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:10,665 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694850665"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694850665"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694850665"}]},"ts":"1679694850665"} 2023-03-24 21:54:10,667 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:10,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:10,824 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 733bfd60f65777910098447a7a228a5c, NAME => 'TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:54:10,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,824 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:54:10,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,825 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,826 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,828 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A 2023-03-24 21:54:10,828 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A 2023-03-24 21:54:10,828 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:10,828 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName A 2023-03-24 21:54:10,829 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:10,829 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,831 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B 2023-03-24 21:54:10,831 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B 2023-03-24 21:54:10,831 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:10,831 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName B 2023-03-24 21:54:10,832 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:10,832 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,833 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C 2023-03-24 21:54:10,833 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C 2023-03-24 21:54:10,834 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:10,834 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName C 2023-03-24 21:54:10,834 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:10,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,835 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,836 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,838 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:54:10,839 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:10,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:54:10,842 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 733bfd60f65777910098447a7a228a5c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67401527, jitterRate=0.004361018538475037}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:54:10,842 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:10,843 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., pid=18, masterSystemTime=1679694850819 2023-03-24 21:54:10,845 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,845 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:10,845 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:10,845 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694850845"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694850845"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694850845"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694850845"}]},"ts":"1679694850845"} 2023-03-24 21:54:10,848 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-24 21:54:10,849 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 in 180 msec 2023-03-24 21:54:10,850 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-24 21:54:10,851 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, ASSIGN in 338 msec 2023-03-24 21:54:10,851 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:54:10,852 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694850851"}]},"ts":"1679694850851"} 2023-03-24 21:54:10,853 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:54:10,856 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:54:10,857 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1830 sec 2023-03-24 21:54:11,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-24 21:54:11,782 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-24 21:54:11,783 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5661dd0b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:11,789 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d849565, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:11,791 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:11,795 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58450, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:11,797 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:54:11,799 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:35794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:54:11,806 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:54:11,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$12(2454): Client=jenkins//172.31.2.10 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:54:11,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:12,228 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:54:12,229 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:54:12,229 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:54:12,233 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 21:54:12,245 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, REOPEN/MOVE}] 2023-03-24 21:54:12,246 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, REOPEN/MOVE 2023-03-24 21:54:12,246 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,247 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694852246"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694852246"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694852246"}]},"ts":"1679694852246"} 2023-03-24 21:54:12,248 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:12,400 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 733bfd60f65777910098447a7a228a5c, disabling compactions & flushes 2023-03-24 21:54:12,401 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. after waiting 0 ms 2023-03-24 21:54:12,401 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 21:54:12,406 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,406 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:12,406 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 733bfd60f65777910098447a7a228a5c to self. 2023-03-24 21:54:12,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,408 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=CLOSED 2023-03-24 21:54:12,408 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694852408"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694852408"}]},"ts":"1679694852408"} 2023-03-24 21:54:12,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-24 21:54:12,410 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 in 161 msec 2023-03-24 21:54:12,411 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=true 2023-03-24 21:54:12,561 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,562 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694852561"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694852561"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694852561"}]},"ts":"1679694852561"} 2023-03-24 21:54:12,563 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:12,718 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 733bfd60f65777910098447a7a228a5c, NAME => 'TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:54:12,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:54:12,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,721 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,722 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A 2023-03-24 21:54:12,722 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A 2023-03-24 21:54:12,722 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:12,725 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName A 2023-03-24 21:54:12,727 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:12,728 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,729 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B 2023-03-24 21:54:12,729 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B 2023-03-24 21:54:12,729 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:12,729 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName B 2023-03-24 21:54:12,730 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:12,730 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,731 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C 2023-03-24 21:54:12,731 DEBUG [StoreOpener-733bfd60f65777910098447a7a228a5c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C 2023-03-24 21:54:12,731 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:12,732 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 733bfd60f65777910098447a7a228a5c columnFamilyName C 2023-03-24 21:54:12,732 INFO [StoreOpener-733bfd60f65777910098447a7a228a5c-1] regionserver.HStore(310): Store=733bfd60f65777910098447a7a228a5c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:12,732 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,733 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,734 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:54:12,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 733bfd60f65777910098447a7a228a5c; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60997833, jitterRate=-0.09106145799160004}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:54:12,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:12,739 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., pid=23, masterSystemTime=1679694852715 2023-03-24 21:54:12,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,740 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,741 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694852740"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694852740"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694852740"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694852740"}]},"ts":"1679694852740"} 2023-03-24 21:54:12,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-24 21:54:12,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 in 179 msec 2023-03-24 21:54:12,745 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-24 21:54:12,745 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, REOPEN/MOVE in 499 msec 2023-03-24 21:54:12,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-24 21:54:12,748 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-03-24 21:54:12,749 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-03-24 21:54:12,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-24 21:54:12,758 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1ed04176 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,763 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79c53329, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,764 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x41982377 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,769 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bc525f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,769 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x2a2c2ab1 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,773 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2aaf2718, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,773 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x642eab2b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,778 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@376ac751, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,778 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5a286c6d to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,782 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bcee3f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,782 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x16549bf9 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,787 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a393ba0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,787 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x462bd7ef to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,791 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74921780, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,791 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1529328e to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,794 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d251b6b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,795 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x50f79162 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:12,799 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c9674b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:12,804 DEBUG [hconnection-0x7087cb0d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,804 DEBUG [hconnection-0x4ce5f918-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,804 DEBUG [hconnection-0x5e5876fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,805 DEBUG [hconnection-0x70b15231-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,806 DEBUG [hconnection-0x3400eb99-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:12,806 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58488, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,806 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58490, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,812 DEBUG [hconnection-0x208044e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,813 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58468, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,813 DEBUG [hconnection-0x42e33ca2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,813 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58466, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,813 DEBUG [hconnection-0x244757e7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,813 DEBUG [hconnection-0x70294668-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:12,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,817 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58494, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,818 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58514, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,822 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58502, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,822 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:58520, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:12,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:12,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:12,823 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:12,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:12,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:12,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:12,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:12,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:12,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:12,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:12,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:12,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:12,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:12,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:12,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:12,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:12,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:12,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:12,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:12,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:12,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:12,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:12,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:12,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:12,833 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:12,833 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 21:54:12,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:12,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:12,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694852859, max timestamp=1679694852877Num uniques 859; 2023-03-24 21:54:12,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:12,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694852859, max timestamp=1679694852877Num uniques 859; 2023-03-24 21:54:12,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:12,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694852859, max timestamp=1679694852877Num uniques 859; 2023-03-24 21:54:12,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:12,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:12,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694912915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694912915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694912916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:12,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694912917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:12,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:12,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694912919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694913019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694913020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694913021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694913021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694913024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694913224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694913224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694913225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694913225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694913229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,379 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324965b7927a34444b58cbc1aa0d0491ea4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324965b7927a34444b58cbc1aa0d0491ea4 2023-03-24 21:54:13,415 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/c65687e07be0415e80523835abb08713 2023-03-24 21:54:13,473 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:54:13,528 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694913527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694913527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694913528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694913529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:13,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694913530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:13,845 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/79333fe6505e4ebcb0e7a54a00421177 2023-03-24 21:54:14,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694914031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:14,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694914032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:14,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694914033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694914033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:14,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694914033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,259 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/708e1e7d123b429d8484fcd1a62119bf 2023-03-24 21:54:14,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/c65687e07be0415e80523835abb08713 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713 2023-03-24 21:54:14,269 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713, entries=150, sequenceid=14, filesize=25.1 K 2023-03-24 21:54:14,270 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/79333fe6505e4ebcb0e7a54a00421177 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177 2023-03-24 21:54:14,275 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 21:54:14,278 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/708e1e7d123b429d8484fcd1a62119bf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf 2023-03-24 21:54:14,282 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 21:54:14,283 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 733bfd60f65777910098447a7a228a5c in 1450ms, sequenceid=14, compaction requested=false 2023-03-24 21:54:14,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:14,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:14,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:14,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:14,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:14,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,285 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:14,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5aecd2f7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:14,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:14,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:14,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:14,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:14,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:14,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:14,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:14,293 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:14,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:14,297 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:14,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:14,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:14,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:14,297 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:14,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,301 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:14,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,312 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,312 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:14,312 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:14,412 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:14,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:14,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:14,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:14,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:14,418 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:14,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:14,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:14,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:14,425 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:14,427 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:14,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:14,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:14,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:14,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:14,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:14,431 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 21:54:14,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:14,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:14,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:14,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:14,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:14,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:14,461 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032485e428addc274d5385ad7e7300a2d502 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032485e428addc274d5385ad7e7300a2d502 2023-03-24 21:54:14,470 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/6ae5b02063294f2a87fd857833aa0a3a 2023-03-24 21:54:14,501 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/c9b0be64278a4da2a55e0bf9bda0c3f3 2023-03-24 21:54:14,538 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/fdbe92db0ed04ba1add9c9f5c26e866b 2023-03-24 21:54:14,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/6ae5b02063294f2a87fd857833aa0a3a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a 2023-03-24 21:54:14,555 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a, entries=150, sequenceid=50, filesize=25.1 K 2023-03-24 21:54:14,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/c9b0be64278a4da2a55e0bf9bda0c3f3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3 2023-03-24 21:54:14,570 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 21:54:14,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/fdbe92db0ed04ba1add9c9f5c26e866b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b 2023-03-24 21:54:14,590 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 21:54:14,591 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 733bfd60f65777910098447a7a228a5c in 160ms, sequenceid=50, compaction requested=false 2023-03-24 21:54:14,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:14,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:14,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:14,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:14,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:14,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,593 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:14,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:14,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eb0a8c0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:14,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:14,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:14,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:14,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:14,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:14,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:14,602 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:14,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:14,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:14,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:14,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:14,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:14,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:14,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:14,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,609 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,610 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:14,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:14,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,615 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:14,615 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:14,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:14,715 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:14,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:14,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:14,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:14,730 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:14,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:14,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:14,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:14,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:14,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:14,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:14,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:14,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:14,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:14,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:14,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:14,737 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,737 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:14,738 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:14,738 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:14,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:14,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:14,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:14,740 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:14,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:14,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57b76198[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:14,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:14,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,744 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:14,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:14,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:14,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:14,746 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:14,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,746 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,746 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:14,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:14,750 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:14,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:14,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:14,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:14,750 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:14,756 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,756 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,756 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:14,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:14,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:14,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,763 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:14,763 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:14,864 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:14,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:14,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:14,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:14,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:14,869 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:14,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:14,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:14,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:14,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:14,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:14,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:14,873 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,873 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:14,873 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:14,873 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:14,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:14,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,876 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,876 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,877 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:14,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b850fa9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:14,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:14,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,884 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:14,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,885 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:14,885 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:14,885 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:14,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:14,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:14,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:14,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:14,890 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:14,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:14,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:14,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:14,890 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:14,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:14,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:14,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:14,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:14,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,895 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:14,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:14,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:14,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:14,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:14,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:14,898 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:14,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:14,898 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:14,999 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:14,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,004 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,007 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,007 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,007 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,008 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,008 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,008 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:15,008 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,011 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47420144[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,016 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,016 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,016 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,018 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,020 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,021 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,022 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,022 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,022 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,029 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,029 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:15,083 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:15,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:15,119 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a3faaa50e27d4527be22edcab44a3c53 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a3faaa50e27d4527be22edcab44a3c53 2023-03-24 21:54:15,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694855109, max timestamp=1679694855122Num uniques 859; 2023-03-24 21:54:15,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694915111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694855109, max timestamp=1679694855122Num uniques 859; 2023-03-24 21:54:15,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:15,126 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:15,128 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694855109, max timestamp=1679694855122Num uniques 859; 2023-03-24 21:54:15,129 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:15,129 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,147 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,149 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=65, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/a38e3c9a918c4ce48bb5f570802bfa40 2023-03-24 21:54:15,150 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694915147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,151 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694915148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694915151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694915151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,159 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,159 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,159 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,159 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,162 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,162 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2cdf57c4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,169 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,169 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,169 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,169 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,173 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,173 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,173 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,179 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,179 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,179 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,188 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,188 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:15,230 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694915227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694915253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694915253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694915255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694915255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,288 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,293 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,304 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40084866[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,307 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,307 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,321 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,321 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:15,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,321 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,422 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,425 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,427 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,429 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,429 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,429 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,430 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,430 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,431 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1323ff89[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694915431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,439 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,439 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,447 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,447 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694915457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694915458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694915458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694915458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,547 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,568 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/3fbd6ec82784489286c5918dd27153c3 2023-03-24 21:54:15,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,570 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,574 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,574 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,584 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,584 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,585 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,593 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@443c8488[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,612 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,612 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:15,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,624 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,624 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,624 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,724 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,728 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,735 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,735 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,735 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,735 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,737 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694915736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@128b8abc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,743 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,746 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,746 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,749 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,755 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,754 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,755 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,755 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694915761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694915761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694915762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,766 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:15,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694915763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,855 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,863 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,863 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,864 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,864 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,864 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,864 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10c2bbf3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,869 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,869 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,869 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,869 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:15,870 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:15,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:15,873 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:15,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:15,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:15,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:15,873 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:15,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,876 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:15,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,883 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:15,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:15,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:15,883 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:15,883 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:15,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:15,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:15,983 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:15,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:15,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:15,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:15,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:15,987 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:15,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:15,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:15,988 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,988 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,989 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:15,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:15,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:15,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:15,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:15,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:15,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:15,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:15,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:15,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:15,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:15,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:15,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:15,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c91d1d5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:15,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:15,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:15,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:15,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:15,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:15,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:16,000 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:16,000 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:16,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,000 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:16,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:16,004 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=65 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d5506e378a0e4176abad0f1c548af160 2023-03-24 21:54:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:16,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:16,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:16,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:16,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:16,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:16,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:16,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:16,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:16,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:16,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:16,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/a38e3c9a918c4ce48bb5f570802bfa40 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40 2023-03-24 21:54:16,012 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:16,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:16,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:16,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:16,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:16,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40, entries=450, sequenceid=65, filesize=65.8 K 2023-03-24 21:54:16,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:16,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/3fbd6ec82784489286c5918dd27153c3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3 2023-03-24 21:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:16,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:16,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,025 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:16,025 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:16,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:16,025 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:16,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:16,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:16,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:16,025 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:16,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:16,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:16,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:16,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,026 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:16,026 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:16,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:16,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:16,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3, entries=150, sequenceid=65, filesize=11.6 K 2023-03-24 21:54:16,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d5506e378a0e4176abad0f1c548af160 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160 2023-03-24 21:54:16,050 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160, entries=150, sequenceid=65, filesize=11.6 K 2023-03-24 21:54:16,052 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 733bfd60f65777910098447a7a228a5c in 969ms, sequenceid=65, compaction requested=true 2023-03-24 21:54:16,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:16,052 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:16,052 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:16,053 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:16,053 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:16,053 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:16,054 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 118886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:16,054 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/A is initiating minor compaction (all files) 2023-03-24 21:54:16,054 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/A in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:16,054 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=116.1 K 2023-03-24 21:54:16,055 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:16,055 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/B is initiating minor compaction (all files) 2023-03-24 21:54:16,055 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/B in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:16,055 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=34.8 K 2023-03-24 21:54:16,055 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c65687e07be0415e80523835abb08713, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694852827 2023-03-24 21:54:16,056 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6ae5b02063294f2a87fd857833aa0a3a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694852913 2023-03-24 21:54:16,056 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 79333fe6505e4ebcb0e7a54a00421177, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694852827 2023-03-24 21:54:16,056 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting a38e3c9a918c4ce48bb5f570802bfa40, keycount=450, bloomtype=ROW, size=65.8 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855071 2023-03-24 21:54:16,056 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c9b0be64278a4da2a55e0bf9bda0c3f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694852913 2023-03-24 21:54:16,057 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3fbd6ec82784489286c5918dd27153c3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855074 2023-03-24 21:54:16,111 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#B#compaction#133 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:16,114 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#A#compaction#132 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:16,126 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:16,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:16,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:16,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:16,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:16,141 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:16,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:16,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:16,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:16,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:16,143 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,143 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:16,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:16,143 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,143 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:16,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:16,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:16,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:16,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:16,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:16,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:16,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:16,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:16,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:16,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:16,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:16,161 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:16,161 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:16,161 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=120.76 KB heapSize=335.77 KB 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:16,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:16,187 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/cf060e620701401f950bba1e6a9b1c45 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cf060e620701401f950bba1e6a9b1c45 2023-03-24 21:54:16,193 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 733bfd60f65777910098447a7a228a5c/A of 733bfd60f65777910098447a7a228a5c into cf060e620701401f950bba1e6a9b1c45(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:16,193 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:16,193 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/A, priority=13, startTime=1679694856052; duration=0sec 2023-03-24 21:54:16,194 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:16,194 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:16,195 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:16,196 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/C is initiating minor compaction (all files) 2023-03-24 21:54:16,196 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/C in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:16,196 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=34.8 K 2023-03-24 21:54:16,196 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 708e1e7d123b429d8484fcd1a62119bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694852827 2023-03-24 21:54:16,197 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting fdbe92db0ed04ba1add9c9f5c26e866b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694852913 2023-03-24 21:54:16,197 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d5506e378a0e4176abad0f1c548af160, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855074 2023-03-24 21:54:16,205 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#C#compaction#135 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:16,235 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/7ef5b61d24574355a444c03b56b3ded6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/7ef5b61d24574355a444c03b56b3ded6 2023-03-24 21:54:16,241 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 733bfd60f65777910098447a7a228a5c/C of 733bfd60f65777910098447a7a228a5c into 7ef5b61d24574355a444c03b56b3ded6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:16,242 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:16,242 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/C, priority=13, startTime=1679694856053; duration=0sec 2023-03-24 21:54:16,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:16,242 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:16,242 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:16,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694916266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694916266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694916268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694916269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694916269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,371 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694916371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,554 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/8de558ff75a24776b7623de03dedc66f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8de558ff75a24776b7623de03dedc66f 2023-03-24 21:54:16,559 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 733bfd60f65777910098447a7a228a5c/B of 733bfd60f65777910098447a7a228a5c into 8de558ff75a24776b7623de03dedc66f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:16,559 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:16,559 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/B, priority=13, startTime=1679694856052; duration=0sec 2023-03-24 21:54:16,559 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:16,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694916572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:16,591 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245b29a5cd55ac468c8f3afa4b59619c3c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245b29a5cd55ac468c8f3afa4b59619c3c 2023-03-24 21:54:16,879 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:16,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694916877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,020 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/1f53daf356d0450390f18d2cdd9681ed 2023-03-24 21:54:17,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:17,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58502 deadline: 1679694917274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:17,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694917274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:17,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694917274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:17,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694917275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:17,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58508 deadline: 1679694917383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,437 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/bd0969d733164b199103550931ba0787 2023-03-24 21:54:17,450 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/cba21da84eba40cba146afcfa016e47e 2023-03-24 21:54:17,455 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/1f53daf356d0450390f18d2cdd9681ed as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed 2023-03-24 21:54:17,459 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed, entries=150, sequenceid=98, filesize=25.1 K 2023-03-24 21:54:17,460 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/bd0969d733164b199103550931ba0787 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787 2023-03-24 21:54:17,464 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 21:54:17,465 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/cba21da84eba40cba146afcfa016e47e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e 2023-03-24 21:54:17,476 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 21:54:17,481 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~335.72 KB/343776, currentSize=60.38 KB/61830 for 733bfd60f65777910098447a7a228a5c in 1320ms, sequenceid=98, compaction requested=false 2023-03-24 21:54:17,481 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:17,481 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:17,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,486 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:17,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:17,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:17,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca91e0b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:17,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:17,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,492 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:17,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:17,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:17,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:17,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:17,494 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:17,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:17,497 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:17,497 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:17,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:17,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:17,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:17,501 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,503 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:17,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:17,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:17,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:17,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,506 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:17,506 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:17,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:17,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:17,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:17,606 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:17,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:17,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:17,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:17,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:17,611 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:17,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:17,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:17,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:17,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:17,613 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:17,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:17,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:17,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:17,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:17,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:17,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:17,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:17,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:17,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:17,631 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:17,631 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:17,631 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:17,631 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:17,631 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:17,632 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:18,090 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241f863ab65ea04c5089a9792de37a2191 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241f863ab65ea04c5089a9792de37a2191 2023-03-24 21:54:18,097 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/6b65f54176124ad3a10dcc3c7fbdaadf 2023-03-24 21:54:18,114 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/ee8eee78549446f2a1a06a1fa2d1c910 2023-03-24 21:54:18,127 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/17d4f2881ac642818a432dbe67237a94 2023-03-24 21:54:18,132 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/6b65f54176124ad3a10dcc3c7fbdaadf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf 2023-03-24 21:54:18,136 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf, entries=100, sequenceid=113, filesize=18.4 K 2023-03-24 21:54:18,137 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/ee8eee78549446f2a1a06a1fa2d1c910 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910 2023-03-24 21:54:18,142 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910, entries=100, sequenceid=113, filesize=9.3 K 2023-03-24 21:54:18,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/17d4f2881ac642818a432dbe67237a94 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94 2023-03-24 21:54:18,149 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94, entries=100, sequenceid=113, filesize=9.3 K 2023-03-24 21:54:18,150 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 733bfd60f65777910098447a7a228a5c in 519ms, sequenceid=113, compaction requested=true 2023-03-24 21:54:18,150 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:18,150 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:18,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:18,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:18,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:18,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:18,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7239bd48[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:18,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:18,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,157 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:18,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:18,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:18,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:18,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:18,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:18,163 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:18,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:18,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:18,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:18,163 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:18,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:18,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:18,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:18,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,174 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:18,174 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:18,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:18,274 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:18,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:18,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:18,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:18,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:18,278 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,281 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:18,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:18,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:18,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:18,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:18,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:18,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:18,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:18,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:18,287 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:18,287 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:18,287 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:18,287 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:18,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:18,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:18,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:18,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:18,289 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,289 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:18,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77219ddf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:18,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:18,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:18,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:18,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:18,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:18,294 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,294 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,294 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:18,294 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:18,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:18,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:18,297 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:18,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:18,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:18,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:18,297 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:18,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:18,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:18,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:18,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:18,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:18,301 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:18,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:18,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:18,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:18,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:18,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:18,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:18,305 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:18,305 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:18,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:18,305 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:18,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:18,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:18,405 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:18,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:18,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:18,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:18,410 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,412 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:18,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:18,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:18,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,413 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:18,413 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:18,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:18,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:18,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:18,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:18,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:18,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:18,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:18,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:18,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:18,431 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 21:54:18,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:18,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:18,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:18,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:18,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:18,432 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:18,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:18,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694858511, max timestamp=1679694858530Num uniques 859; 2023-03-24 21:54:18,531 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694858511, max timestamp=1679694858530Num uniques 859; 2023-03-24 21:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694858511, max timestamp=1679694858530Num uniques 859; 2023-03-24 21:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:18,532 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694858556, max timestamp=1679694858579Num uniques 1068; 2023-03-24 21:54:18,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 21:54:18,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694858556, max timestamp=1679694858579Num uniques 1068; 2023-03-24 21:54:18,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 21:54:18,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694858556, max timestamp=1679694858579Num uniques 1068; 2023-03-24 21:54:18,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694858606, max timestamp=1679694858637Num uniques 1077; 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694858606, max timestamp=1679694858637Num uniques 1077; 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694858606, max timestamp=1679694858637Num uniques 1077; 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:54:18,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679694858685, max timestamp=1679694858700Num uniques 1136; 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679694858685, max timestamp=1679694858700Num uniques 1136; 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679694858685, max timestamp=1679694858700Num uniques 1136; 2023-03-24 21:54:18,703 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=10806, suffix heap size=146112,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=236 2023-03-24 21:54:18,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694858740, max timestamp=1679694858755Num uniques 1095; 2023-03-24 21:54:18,759 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694858740, max timestamp=1679694858755Num uniques 1095; 2023-03-24 21:54:18,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694858740, max timestamp=1679694858755Num uniques 1095; 2023-03-24 21:54:18,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 21:54:18,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 21:54:18,760 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694858815, max timestamp=1679694858828Num uniques 1054; 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694858815, max timestamp=1679694858828Num uniques 1054; 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694858815, max timestamp=1679694858828Num uniques 1054; 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:54:18,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:54:18,863 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032409c414926d58465983d5a20343fa8ec3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032409c414926d58465983d5a20343fa8ec3 2023-03-24 21:54:18,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694858860, max timestamp=1679694858880Num uniques 1013; 2023-03-24 21:54:18,882 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 21:54:18,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694858860, max timestamp=1679694858880Num uniques 1013; 2023-03-24 21:54:18,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694858860, max timestamp=1679694858880Num uniques 1013; 2023-03-24 21:54:18,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 21:54:18,883 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=9748, suffix heap size=133320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=213 2023-03-24 21:54:18,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694858922, max timestamp=1679694858935Num uniques 1072; 2023-03-24 21:54:18,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 21:54:18,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679694858922, max timestamp=1679694858935Num uniques 1072; 2023-03-24 21:54:18,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 21:54:18,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1679694858922, max timestamp=1679694858935Num uniques 1072; 2023-03-24 21:54:18,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694858954, max timestamp=1679694858983Num uniques 1081; 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694858954, max timestamp=1679694858983Num uniques 1031; 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=10576, suffix heap size=140392,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=231 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694858954, max timestamp=1679694858983Num uniques 1031; 2023-03-24 21:54:18,986 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-03-24 21:54:19,027 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679694859004, max timestamp=1679694859025Num uniques 1090; 2023-03-24 21:54:19,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-24 21:54:19,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679694859004, max timestamp=1679694859025Num uniques 1090; 2023-03-24 21:54:19,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1679694859004, max timestamp=1679694859025Num uniques 1090; 2023-03-24 21:54:19,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-24 21:54:19,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-03-24 21:54:19,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679694859050, max timestamp=1679694859068Num uniques 1099; 2023-03-24 21:54:19,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679694859050, max timestamp=1679694859068Num uniques 1099; 2023-03-24 21:54:19,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-24 21:54:19,071 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679694859050, max timestamp=1679694859068Num uniques 1099; 2023-03-24 21:54:19,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-24 21:54:19,072 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-24 21:54:19,123 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679694859093, max timestamp=1679694859121Num uniques 1108; 2023-03-24 21:54:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679694859093, max timestamp=1679694859121Num uniques 1108; 2023-03-24 21:54:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1679694859093, max timestamp=1679694859121Num uniques 1108; 2023-03-24 21:54:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 21:54:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 21:54:19,124 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=7230, suffix heap size=143200,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=158 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679694859161, max timestamp=1679694859179Num uniques 1017; 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679694859161, max timestamp=1679694859179Num uniques 1017; 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679694859161, max timestamp=1679694859179Num uniques 1017; 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-24 21:54:19,182 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-03-24 21:54:19,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679694859205, max timestamp=1679694859227Num uniques 1076; 2023-03-24 21:54:19,229 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-24 21:54:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679694859205, max timestamp=1679694859227Num uniques 1076; 2023-03-24 21:54:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679694859205, max timestamp=1679694859227Num uniques 1076; 2023-03-24 21:54:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-24 21:54:19,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-03-24 21:54:19,270 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=118, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/ddb97f82e25a4ef5a50cd0321c8ef9f7 2023-03-24 21:54:19,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694859254, max timestamp=1679694859269Num uniques 1035; 2023-03-24 21:54:19,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-24 21:54:19,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694859254, max timestamp=1679694859269Num uniques 1035; 2023-03-24 21:54:19,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-03-24 21:54:19,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=185, cellSize=8470, totalHeapSize=19480, min timestamp=1679694859254, max timestamp=1679694859269Num uniques 1035; 2023-03-24 21:54:19,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=8470, suffix heap size=135608,new segment heap size=19480 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=185 2023-03-24 21:54:19,282 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/9a24a4906ead4e8ab1f0dc62d6e03cee 2023-03-24 21:54:19,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1679694859308, max timestamp=1679694859317Num uniques 1094; 2023-03-24 21:54:19,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-24 21:54:19,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679694859308, max timestamp=1679694859317Num uniques 1094; 2023-03-24 21:54:19,319 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-03-24 21:54:19,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1679694859308, max timestamp=1679694859317Num uniques 1044; 2023-03-24 21:54:19,320 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=47814, new segment data size=11174, suffix heap size=136544,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1044, new segment cells count=244 2023-03-24 21:54:19,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=453, cellSize=20745, totalHeapSize=47352, min timestamp=1679694859312, max timestamp=1679694859336Num uniques 1153; 2023-03-24 21:54:19,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=20745, suffix heap size=147880,new segment heap size=47352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=453 2023-03-24 21:54:19,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=453, cellSize=20745, totalHeapSize=47352, min timestamp=1679694859312, max timestamp=1679694859336Num uniques 1103; 2023-03-24 21:54:19,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=20745, suffix heap size=142680,new segment heap size=47352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=453 2023-03-24 21:54:19,338 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=453, cellSize=20745, totalHeapSize=47352, min timestamp=1679694859312, max timestamp=1679694859336Num uniques 1103; 2023-03-24 21:54:19,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=20745, suffix heap size=142680,new segment heap size=47352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=453 2023-03-24 21:54:19,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694919352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=412, cellSize=18862, totalHeapSize=43088, min timestamp=1679694859343, max timestamp=1679694859353Num uniques 1312; 2023-03-24 21:54:19,355 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=60082, new segment data size=18862, suffix heap size=164416,new segment heap size=43088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1312, new segment cells count=412 2023-03-24 21:54:19,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=412, cellSize=18862, totalHeapSize=43088, min timestamp=1679694859343, max timestamp=1679694859353Num uniques 1312; 2023-03-24 21:54:19,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=60082, new segment data size=18862, suffix heap size=164416,new segment heap size=43088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1312, new segment cells count=412 2023-03-24 21:54:19,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=412, cellSize=18862, totalHeapSize=43088, min timestamp=1679694859343, max timestamp=1679694859353Num uniques 1312; 2023-03-24 21:54:19,356 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=60082, new segment data size=18862, suffix heap size=164416,new segment heap size=43088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1312, new segment cells count=412 2023-03-24 21:54:19,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=421, cellSize=19276, totalHeapSize=44024, min timestamp=1679694859365, max timestamp=1679694859374Num uniques 1271; 2023-03-24 21:54:19,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=58206, new segment data size=19276, suffix heap size=160152,new segment heap size=44024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1271, new segment cells count=421 2023-03-24 21:54:19,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694919374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694919374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694859368, max timestamp=1679694859374Num uniques 1271; 2023-03-24 21:54:19,379 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=58206, new segment data size=6870, suffix heap size=160152,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1271, new segment cells count=150 2023-03-24 21:54:19,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=421, cellSize=19276, totalHeapSize=44024, min timestamp=1679694859365, max timestamp=1679694859374Num uniques 1271; 2023-03-24 21:54:19,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=58206, new segment data size=19276, suffix heap size=160152,new segment heap size=44024 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1271, new segment cells count=421 2023-03-24 21:54:19,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679694859400, max timestamp=1679694859406Num uniques 1009; 2023-03-24 21:54:19,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8240, suffix heap size=132904,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=180 2023-03-24 21:54:19,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679694859400, max timestamp=1679694859406Num uniques 1280; 2023-03-24 21:54:19,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=58620, new segment data size=8240, suffix heap size=161088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1280, new segment cells count=180 2023-03-24 21:54:19,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=180, cellSize=8240, totalHeapSize=18960, min timestamp=1679694859400, max timestamp=1679694859406Num uniques 1280; 2023-03-24 21:54:19,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=58620, new segment data size=8240, suffix heap size=161088,new segment heap size=18960 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1280, new segment cells count=180 2023-03-24 21:54:19,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679694859425, max timestamp=1679694859432Num uniques 1039; 2023-03-24 21:54:19,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679694859425, max timestamp=1679694859432Num uniques 1039; 2023-03-24 21:54:19,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-03-24 21:54:19,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-03-24 21:54:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679694859425, max timestamp=1679694859432Num uniques 1039; 2023-03-24 21:54:19,436 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47584, new segment data size=13234, suffix heap size=136024,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1039, new segment cells count=289 2023-03-24 21:54:19,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679694859446, max timestamp=1679694859459Num uniques 1148; 2023-03-24 21:54:19,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=15938, suffix heap size=147360,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=348 2023-03-24 21:54:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679694859446, max timestamp=1679694859459Num uniques 1148; 2023-03-24 21:54:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679694859446, max timestamp=1679694859459Num uniques 1148; 2023-03-24 21:54:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=15938, suffix heap size=147360,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=348 2023-03-24 21:54:19,462 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=15938, suffix heap size=147360,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=348 2023-03-24 21:54:19,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694859474, max timestamp=1679694859481Num uniques 1207; 2023-03-24 21:54:19,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694919481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=11765, suffix heap size=153496,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=257 2023-03-24 21:54:19,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694859474, max timestamp=1679694859481Num uniques 1207; 2023-03-24 21:54:19,483 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=11765, suffix heap size=153496,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=257 2023-03-24 21:54:19,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694859474, max timestamp=1679694859481Num uniques 1207; 2023-03-24 21:54:19,484 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=55275, new segment data size=11765, suffix heap size=153496,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1207, new segment cells count=257 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=516, cellSize=23626, totalHeapSize=53904, min timestamp=1679694859482, max timestamp=1679694859495Num uniques 1116; 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=23626, suffix heap size=144032,new segment heap size=53904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=516 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=516, cellSize=23626, totalHeapSize=53904, min timestamp=1679694859482, max timestamp=1679694859495Num uniques 1116; 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=23626, suffix heap size=144032,new segment heap size=53904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=516 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=516, cellSize=23626, totalHeapSize=53904, min timestamp=1679694859482, max timestamp=1679694859495Num uniques 1116; 2023-03-24 21:54:19,497 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=23626, suffix heap size=144032,new segment heap size=53904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=516 2023-03-24 21:54:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694919511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,514 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694919513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8010, totalHeapSize=18440, min timestamp=1679694859509, max timestamp=1679694859512Num uniques 1375; 2023-03-24 21:54:19,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=62970, new segment data size=8010, suffix heap size=170968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1375, new segment cells count=175 2023-03-24 21:54:19,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8000, totalHeapSize=18440, min timestamp=1679694859509, max timestamp=1679694859512Num uniques 1375; 2023-03-24 21:54:19,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=175, cellSize=8000, totalHeapSize=18440, min timestamp=1679694859509, max timestamp=1679694859512Num uniques 1375; 2023-03-24 21:54:19,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=62960, new segment data size=8000, suffix heap size=170968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1375, new segment cells count=175 2023-03-24 21:54:19,515 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=62960, new segment data size=8000, suffix heap size=170968,new segment heap size=18440 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1375, new segment cells count=175 2023-03-24 21:54:19,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=12994, totalHeapSize=29776, min timestamp=1679694859530, max timestamp=1679694859545Num uniques 1034; 2023-03-24 21:54:19,547 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=47344, new segment data size=12994, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-03-24 21:54:19,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=12995, totalHeapSize=29776, min timestamp=1679694859530, max timestamp=1679694859545Num uniques 1034; 2023-03-24 21:54:19,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47345, new segment data size=12995, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-03-24 21:54:19,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=284, cellSize=12994, totalHeapSize=29776, min timestamp=1679694859530, max timestamp=1679694859545Num uniques 1034; 2023-03-24 21:54:19,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=47344, new segment data size=12994, suffix heap size=135504,new segment heap size=29776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1034, new segment cells count=284 2023-03-24 21:54:19,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679694859580, max timestamp=1679694859593Num uniques 1143; 2023-03-24 21:54:19,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=8838, suffix heap size=146840,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=193 2023-03-24 21:54:19,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679694859580, max timestamp=1679694859593Num uniques 1143; 2023-03-24 21:54:19,595 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=8838, suffix heap size=146840,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=193 2023-03-24 21:54:19,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=193, cellSize=8838, totalHeapSize=20312, min timestamp=1679694859580, max timestamp=1679694859593Num uniques 1143; 2023-03-24 21:54:19,596 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=52348, new segment data size=8838, suffix heap size=146840,new segment heap size=20312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1143, new segment cells count=193 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1679694859608, max timestamp=1679694859618Num uniques 1052; 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1679694859608, max timestamp=1679694859618Num uniques 1052; 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=352, cellSize=16120, totalHeapSize=36848, min timestamp=1679694859608, max timestamp=1679694859618Num uniques 1052; 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=16120, suffix heap size=137376,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=352 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=16120, suffix heap size=137376,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=352 2023-03-24 21:54:19,621 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48180, new segment data size=16120, suffix heap size=137376,new segment heap size=36848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1052, new segment cells count=352 2023-03-24 21:54:19,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1679694859629, max timestamp=1679694859645Num uniques 1211; 2023-03-24 21:54:19,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=14236, suffix heap size=153912,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=311 2023-03-24 21:54:19,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1679694859629, max timestamp=1679694859645Num uniques 1211; 2023-03-24 21:54:19,648 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=14236, suffix heap size=153912,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=311 2023-03-24 21:54:19,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=311, cellSize=14236, totalHeapSize=32584, min timestamp=1679694859629, max timestamp=1679694859645Num uniques 1211; 2023-03-24 21:54:19,650 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=55456, new segment data size=14236, suffix heap size=153912,new segment heap size=32584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1211, new segment cells count=311 2023-03-24 21:54:19,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1679694859653, max timestamp=1679694859670Num uniques 1170; 2023-03-24 21:54:19,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1679694859653, max timestamp=1679694859670Num uniques 1170; 2023-03-24 21:54:19,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-03-24 21:54:19,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-03-24 21:54:19,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=270, cellSize=12360, totalHeapSize=28320, min timestamp=1679694859653, max timestamp=1679694859670Num uniques 1170; 2023-03-24 21:54:19,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53580, new segment data size=12360, suffix heap size=149648,new segment heap size=28320 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1170, new segment cells count=270 2023-03-24 21:54:19,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679694859685, max timestamp=1679694859690Num uniques 1129; 2023-03-24 21:54:19,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=8194, suffix heap size=145384,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=179 2023-03-24 21:54:19,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=329, cellSize=15064, totalHeapSize=34456, min timestamp=1679694859684, max timestamp=1679694859690Num uniques 1129; 2023-03-24 21:54:19,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=179, cellSize=8194, totalHeapSize=18856, min timestamp=1679694859685, max timestamp=1679694859690Num uniques 1129; 2023-03-24 21:54:19,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=8194, suffix heap size=145384,new segment heap size=18856 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=179 2023-03-24 21:54:19,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=51704, new segment data size=15064, suffix heap size=145384,new segment heap size=34456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1129, new segment cells count=329 2023-03-24 21:54:19,698 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/718f526a818e4fc99f907d36942fc88b 2023-03-24 21:54:19,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/ddb97f82e25a4ef5a50cd0321c8ef9f7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7 2023-03-24 21:54:19,709 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7, entries=50, sequenceid=118, filesize=11.6 K 2023-03-24 21:54:19,710 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/9a24a4906ead4e8ab1f0dc62d6e03cee as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee 2023-03-24 21:54:19,716 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee, entries=50, sequenceid=118, filesize=7.0 K 2023-03-24 21:54:19,717 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/718f526a818e4fc99f907d36942fc88b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b 2023-03-24 21:54:19,722 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b, entries=50, sequenceid=118, filesize=7.0 K 2023-03-24 21:54:19,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694859708, max timestamp=1679694859720Num uniques 1188; 2023-03-24 21:54:19,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=54408, new segment data size=8608, suffix heap size=151520,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1188, new segment cells count=188 2023-03-24 21:54:19,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694859708, max timestamp=1679694859720Num uniques 1038; 2023-03-24 21:54:19,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-03-24 21:54:19,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694859708, max timestamp=1679694859720Num uniques 1038; 2023-03-24 21:54:19,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47538, new segment data size=8608, suffix heap size=135920,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1038, new segment cells count=188 2023-03-24 21:54:19,730 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=26.84 KB/27480 for 733bfd60f65777910098447a7a228a5c in 1299ms, sequenceid=118, compaction requested=true 2023-03-24 21:54:19,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:19,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:19,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:19,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:19,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:19,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:19,733 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=53.67 KB heapSize=143.20 KB 2023-03-24 21:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:19,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:19,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:19,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652ce597[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:19,736 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:19,736 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:19,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:19,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:19,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:19,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:19,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:19,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:19,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:19,778 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:19,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:19,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:19,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:19,778 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:19,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:19,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:19,781 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:19,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:19,787 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:19,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:19,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:19,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:19,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:19,788 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:19,788 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:19,789 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241694fe1c428143ae923817c0c9ee9bee to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241694fe1c428143ae923817c0c9ee9bee 2023-03-24 21:54:19,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694859782, max timestamp=1679694859795Num uniques 859; 2023-03-24 21:54:19,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:19,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694859782, max timestamp=1679694859795Num uniques 859; 2023-03-24 21:54:19,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:19,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694859782, max timestamp=1679694859795Num uniques 859; 2023-03-24 21:54:19,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:19,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694919809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694919810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694919813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694919815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694919814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,888 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:19,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:19,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:19,892 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:19,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:19,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:19,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:19,894 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,894 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:19,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:19,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:19,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:19,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:19,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:19,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:19,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:19,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:19,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:19,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:19,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:19,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:19,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:19,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:19,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:19,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:19,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1181e67f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:19,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:19,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:19,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:19,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:19,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:19,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:19,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:19,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:19,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:19,911 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:19,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:19,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:19,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:19,911 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:19,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:19,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:19,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:19,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:19,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,917 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:19,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:19,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694919918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694919918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:19,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:19,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:19,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:19,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:19,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,920 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:19,920 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:19,920 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:19,920 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:19,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:19,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:19,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:19,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694919921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694919921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:19,927 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:19,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694919922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,020 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,024 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,027 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,027 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,028 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,028 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,028 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,028 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:20,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:20,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:20,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:20,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e6dcafa[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:20,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:20,035 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:20,035 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:20,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:20,036 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:20,036 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,036 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:20,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:20,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:20,039 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:20,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:20,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:20,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:20,039 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:20,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:20,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,045 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:20,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,046 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:20,046 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:20,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694920119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694920120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694920128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,131 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694920129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694920129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,146 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,151 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,163 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,163 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,163 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,163 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:20,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:20,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,165 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,166 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,166 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:20,168 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5559658d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:20,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:20,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,169 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,169 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:20,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:20,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:20,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:20,171 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:20,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:20,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:20,175 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:20,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:20,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:20,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:20,175 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:20,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:20,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:20,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:20,188 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:20,188 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:20,201 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=675, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/7703bdee9d52400ebb345e07a8111692 2023-03-24 21:54:20,288 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,293 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,295 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,296 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,296 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:20,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:20,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:20,298 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:20,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,299 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,299 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:20,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57db22ec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:20,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:20,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,305 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:20,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:20,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:20,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:20,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:20,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:20,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:20,310 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:20,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:20,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:20,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:20,311 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:20,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:20,318 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:20,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:20,318 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:20,318 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:20,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,418 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,424 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694920422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,425 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,426 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,426 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,426 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:20,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:20,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694920425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:20,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49778476[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:20,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:20,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,431 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:20,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:20,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:20,432 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:20,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:20,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:20,433 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694920433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694920434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694920434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:20,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:20,436 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:20,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:20,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:20,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:20,436 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:20,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:20,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:20,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,444 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:20,444 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:20,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:20,544 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,547 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,548 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:20,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:20,553 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:20,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@90dadb8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:20,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:20,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,557 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:20,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:20,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:20,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:20,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:20,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:20,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:20,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:20,561 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:20,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:20,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:20,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:20,562 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:20,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:20,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:20,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:20,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:20,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:20,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:20,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:20,574 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:20,574 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:20,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:20,613 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/98603c55d5614f568340b95af5badd1c 2023-03-24 21:54:20,627 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d5f21bf1dae14e19879519f7b8e932a7 2023-03-24 21:54:20,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/7703bdee9d52400ebb345e07a8111692 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692 2023-03-24 21:54:20,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692, entries=400, sequenceid=675, filesize=59.8 K 2023-03-24 21:54:20,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/98603c55d5614f568340b95af5badd1c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c 2023-03-24 21:54:20,640 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c, entries=150, sequenceid=675, filesize=11.9 K 2023-03-24 21:54:20,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d5f21bf1dae14e19879519f7b8e932a7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7 2023-03-24 21:54:20,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7, entries=150, sequenceid=675, filesize=11.9 K 2023-03-24 21:54:20,646 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~143.16 KB/146592, currentSize=134.18 KB/137400 for 733bfd60f65777910098447a7a228a5c in 912ms, sequenceid=675, compaction requested=true 2023-03-24 21:54:20,646 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:20,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:20,646 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:20,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:20,646 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:20,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:20,648 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 143503 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:20,648 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/A is initiating minor compaction (all files) 2023-03-24 21:54:20,648 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 52803 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:20,648 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/A in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,648 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/B is initiating minor compaction (all files) 2023-03-24 21:54:20,648 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cf060e620701401f950bba1e6a9b1c45, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=140.1 K 2023-03-24 21:54:20,648 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/B in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,648 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8de558ff75a24776b7623de03dedc66f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=51.6 K 2023-03-24 21:54:20,648 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cf060e620701401f950bba1e6a9b1c45, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855074 2023-03-24 21:54:20,649 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8de558ff75a24776b7623de03dedc66f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855074 2023-03-24 21:54:20,649 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1f53daf356d0450390f18d2cdd9681ed, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694855145 2023-03-24 21:54:20,649 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bd0969d733164b199103550931ba0787, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694855145 2023-03-24 21:54:20,649 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6b65f54176124ad3a10dcc3c7fbdaadf, keycount=100, bloomtype=ROW, size=18.4 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694856259 2023-03-24 21:54:20,649 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ee8eee78549446f2a1a06a1fa2d1c910, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694856259 2023-03-24 21:54:20,650 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ddb97f82e25a4ef5a50cd0321c8ef9f7, keycount=50, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694858420 2023-03-24 21:54:20,650 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9a24a4906ead4e8ab1f0dc62d6e03cee, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694858420 2023-03-24 21:54:20,650 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7703bdee9d52400ebb345e07a8111692, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859708 2023-03-24 21:54:20,650 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 98603c55d5614f568340b95af5badd1c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859719 2023-03-24 21:54:20,668 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#B#compaction#148 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:20,671 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#A#compaction#147 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:20,674 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:20,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:20,691 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:20,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:20,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:20,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:20,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:20,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:20,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:20,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:20,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:20,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:20,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:20,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:20,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:20,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:20,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:20,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:20,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:20,699 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-03-24 21:54:20,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:20,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:20,700 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:20,700 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:20,700 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:20,700 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:20,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:20,932 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:20,944 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694920942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694920942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694920943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694920944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:20,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:20,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694920944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694921045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694921047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694921047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694921047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694921047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,089 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/16a51dcab9ec494494a5b96017b3791e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/16a51dcab9ec494494a5b96017b3791e 2023-03-24 21:54:21,093 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/B of 733bfd60f65777910098447a7a228a5c into 16a51dcab9ec494494a5b96017b3791e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:21,094 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:21,094 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/B, priority=11, startTime=1679694860646; duration=0sec 2023-03-24 21:54:21,094 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:21,094 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:21,095 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 52803 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:21,096 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/C is initiating minor compaction (all files) 2023-03-24 21:54:21,096 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/C in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:21,096 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/7ef5b61d24574355a444c03b56b3ded6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=51.6 K 2023-03-24 21:54:21,096 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7ef5b61d24574355a444c03b56b3ded6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=65, earliestPutTs=1679694855074 2023-03-24 21:54:21,097 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting cba21da84eba40cba146afcfa016e47e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694855145 2023-03-24 21:54:21,097 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 17d4f2881ac642818a432dbe67237a94, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694856259 2023-03-24 21:54:21,098 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 718f526a818e4fc99f907d36942fc88b, keycount=50, bloomtype=ROW, size=7.0 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694858420 2023-03-24 21:54:21,098 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d5f21bf1dae14e19879519f7b8e932a7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859719 2023-03-24 21:54:21,110 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#C#compaction#150 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:21,124 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248ba27615060341debbe917a955425b54 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ba27615060341debbe917a955425b54 2023-03-24 21:54:21,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694921250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694921250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694921251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694921251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694921252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,528 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/53d66b971c33448aa0d4a145b6a525c4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/53d66b971c33448aa0d4a145b6a525c4 2023-03-24 21:54:21,530 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/6d7bf067e9934999b1b8a9b19ff1de36 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6d7bf067e9934999b1b8a9b19ff1de36 2023-03-24 21:54:21,532 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=709, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/13fc2a2f248e4cb5a4da4a21a67cdcc4 2023-03-24 21:54:21,536 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/C of 733bfd60f65777910098447a7a228a5c into 53d66b971c33448aa0d4a145b6a525c4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:21,536 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:21,536 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/C, priority=11, startTime=1679694860646; duration=0sec 2023-03-24 21:54:21,538 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:21,546 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/A of 733bfd60f65777910098447a7a228a5c into 6d7bf067e9934999b1b8a9b19ff1de36(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:21,546 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:21,546 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/A, priority=11, startTime=1679694860646; duration=0sec 2023-03-24 21:54:21,547 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:21,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694921554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694921554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694921554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,556 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694921556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:21,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694921557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:21,976 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/8880541a3a3a40418e6fd21792e1b428 2023-03-24 21:54:22,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:22,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694922055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:22,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:22,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694922058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694922058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:22,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694922059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:22,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58490 deadline: 1679694922064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,390 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=709 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/03d99e9e5a97489ca164bc5a824f386a 2023-03-24 21:54:22,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/13fc2a2f248e4cb5a4da4a21a67cdcc4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4 2023-03-24 21:54:22,401 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4, entries=150, sequenceid=709, filesize=25.4 K 2023-03-24 21:54:22,402 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/8880541a3a3a40418e6fd21792e1b428 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428 2023-03-24 21:54:22,407 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428, entries=150, sequenceid=709, filesize=11.9 K 2023-03-24 21:54:22,408 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/03d99e9e5a97489ca164bc5a824f386a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a 2023-03-24 21:54:22,414 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a, entries=150, sequenceid=709, filesize=11.9 K 2023-03-24 21:54:22,415 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 733bfd60f65777910098447a7a228a5c in 1716ms, sequenceid=709, compaction requested=false 2023-03-24 21:54:22,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:22,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:22,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:22,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:22,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,420 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:22,420 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:22,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:22,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,423 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:22,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:22,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1766d419[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:22,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:22,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:22,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:22,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:22,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:22,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:22,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:22,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:22,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:22,444 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:22,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:22,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:22,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:22,444 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:22,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:22,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:22,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:22,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:22,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:22,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:22,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:22,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:22,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:22,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:22,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:22,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:22,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:22,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:22,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:22,457 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:22,457 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:22,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:22,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:22,557 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:22,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:22,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:22,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:22,561 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:22,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:22,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:22,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:22,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:22,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:22,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:22,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:22,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:22,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:22,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:22,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:22,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:22,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:22,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:22,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:22,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:22,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:22,565 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:22,565 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:22,587 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032440faf0a979054ce3ae8c4eb3af09150a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032440faf0a979054ce3ae8c4eb3af09150a 2023-03-24 21:54:23,004 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=723, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/bbbc745e1b20401fb407449cf54a0ced 2023-03-24 21:54:23,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:23,063 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694923078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694923078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694923078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694863072, max timestamp=1679694863080Num uniques 859; 2023-03-24 21:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694863072, max timestamp=1679694863080Num uniques 859; 2023-03-24 21:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:23,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694863072, max timestamp=1679694863080Num uniques 859; 2023-03-24 21:54:23,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:54:23,107 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694923107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694923108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694923183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694923183, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,186 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694923184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,210 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 424 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694923208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694923211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694923388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694923388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694923388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 426 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694923412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,416 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/587ae11c2ba84052ab5b4528b5c873c6 2023-03-24 21:54:23,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694923415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694923693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694923694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694923694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,718 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 428 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694923717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:23,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58488 deadline: 1679694923720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,827 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=723 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/e44bb730f6f04219a275027d63fdd9ca 2023-03-24 21:54:23,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/bbbc745e1b20401fb407449cf54a0ced as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced 2023-03-24 21:54:23,836 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced, entries=150, sequenceid=723, filesize=25.4 K 2023-03-24 21:54:23,837 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/587ae11c2ba84052ab5b4528b5c873c6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6 2023-03-24 21:54:23,841 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6, entries=150, sequenceid=723, filesize=11.9 K 2023-03-24 21:54:23,842 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/e44bb730f6f04219a275027d63fdd9ca as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca 2023-03-24 21:54:23,846 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca, entries=150, sequenceid=723, filesize=11.9 K 2023-03-24 21:54:23,847 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 733bfd60f65777910098447a7a228a5c in 1282ms, sequenceid=723, compaction requested=true 2023-03-24 21:54:23,847 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:23,847 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:23,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:23,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:23,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:23,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:23,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,849 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:23,849 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,849 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:23,851 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1924585b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:23,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:23,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:23,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:23,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:23,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:23,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:23,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:23,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:23,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:23,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:23,857 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:23,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:23,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:23,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:23,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:23,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:23,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:23,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:23,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:23,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:23,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:23,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:23,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:23,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:23,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:23,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:23,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:23,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:23,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:23,866 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:23,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,866 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:23,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:23,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:23,966 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:23,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:23,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:23,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:23,976 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:23,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:23,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:23,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:23,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:23,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:23,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:23,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:23,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:23,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:23,981 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:23,981 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:23,982 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:23,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:23,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:23,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:23,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:23,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:23,988 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:23,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:24,004 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032458838345b7794f0899c8d159ed58407a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032458838345b7794f0899c8d159ed58407a 2023-03-24 21:54:24,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:24,199 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:24,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694924205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694924206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,209 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694924207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 430 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694924222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,229 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58488 deadline: 1679694924224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694924309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:58468 deadline: 1679694924310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694924311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,409 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=759, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/8c8779469609491f8a04181839cc98ca 2023-03-24 21:54:24,437 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/c026c9f8deb34991904a7f6f0a065fff 2023-03-24 21:54:24,448 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/e807c92cb1634a69a7b982209a6a7f1a 2023-03-24 21:54:24,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/8c8779469609491f8a04181839cc98ca as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca 2023-03-24 21:54:24,455 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca, entries=150, sequenceid=759, filesize=25.4 K 2023-03-24 21:54:24,456 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/c026c9f8deb34991904a7f6f0a065fff as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff 2023-03-24 21:54:24,460 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff, entries=150, sequenceid=759, filesize=11.9 K 2023-03-24 21:54:24,463 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/e807c92cb1634a69a7b982209a6a7f1a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a 2023-03-24 21:54:24,467 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a, entries=150, sequenceid=759, filesize=11.9 K 2023-03-24 21:54:24,468 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 733bfd60f65777910098447a7a228a5c in 480ms, sequenceid=759, compaction requested=true 2023-03-24 21:54:24,468 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:24,468 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:24,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:24,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:24,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:24,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,470 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:24,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c852b3d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:24,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:24,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,473 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,473 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:24,473 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:24,473 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:24,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:24,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,474 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,474 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,474 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:24,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:24,477 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:24,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:24,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:24,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:24,477 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:24,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,480 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,488 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:24,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:24,488 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,489 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:24,489 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:24,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:24,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:24,516 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:24,529 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242b1bc00c88314d029bf389531207e593 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242b1bc00c88314d029bf389531207e593 2023-03-24 21:54:24,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694924557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694924557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694924558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,589 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:24,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:24,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:24,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:24,592 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:24,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:24,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:24,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:24,597 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:24,597 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:24,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:24,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:24,598 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,598 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:24,598 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:24,598 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:24,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:24,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,600 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:24,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8812b78[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:24,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:24,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,603 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:24,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:24,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:24,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,604 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:24,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:24,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:24,607 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:24,607 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:24,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:24,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:24,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:24,608 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:24,610 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,610 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,610 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,610 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:24,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:24,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,615 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:24,615 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:24,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694924663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694924663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,667 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694924663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,716 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:24,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:24,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:24,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:24,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:24,719 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:24,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:24,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:24,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,721 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,722 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,722 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:24,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:24,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:24,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:24,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:24,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:24,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:24,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:24,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:24,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:24,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:24,729 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:24,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:24,731 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2966f06d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:24,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:24,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:24,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:24,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:24,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:24,734 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:24,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:24,738 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:24,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:24,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:24,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:24,738 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:24,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,748 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:24,748 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:24,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,748 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,748 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,748 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,848 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:24,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:24,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:24,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:24,852 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:24,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:24,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:24,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:24,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:24,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:24,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:24,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:24,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:24,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:24,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:24,858 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:24,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:24,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@123b939a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:24,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:24,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:24,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:24,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:24,862 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:24,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:24,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:24,865 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:24,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:24,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:24,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:24,865 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:24,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694924868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694924868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,871 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:24,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694924868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:24,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:24,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,872 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,872 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:24,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,947 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=772, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/1937bd20bb3943e29cde846362582a98 2023-03-24 21:54:24,972 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:24,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:24,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/950ba3d54aa149b2ac4a168f0410dab9 2023-03-24 21:54:24,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:24,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:24,976 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:24,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:24,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:24,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:24,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:24,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:24,979 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,979 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:24,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:24,980 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:24,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:24,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:24,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:24,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:24,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:24,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:24,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:24,981 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,981 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:24,982 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:24,982 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:24,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:24,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:24,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:24,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:24,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:24,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57a29e23[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:24,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:24,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:24,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:24,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:24,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:24,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:24,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:24,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:24,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:24,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:24,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:24,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:24,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:24,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:24,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:24,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:24,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:24,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,999 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:25,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,002 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,002 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:25,002 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:25,103 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:25,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:25,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:25,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:25,106 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:25,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:25,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:25,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,110 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,110 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:25,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:25,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:25,111 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,111 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:25,111 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:25,111 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:25,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:25,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:25,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:25,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,115 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5679b544[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:25,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:25,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:25,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:25,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:25,118 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:25,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:25,121 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:25,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:25,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:25,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:25,121 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:25,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:25,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,129 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:25,129 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:25,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694925173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694925174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694925174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 432 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694925226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,230 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:25,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:25,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:25,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:25,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:25,233 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:25,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:25,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:25,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,235 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:25,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:25,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:25,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:25,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:25,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:25,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:25,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:25,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:25,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,238 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:25,238 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:25,238 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:25,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:25,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:25,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:25,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:25,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:25,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4882d27e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:25,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58488 deadline: 1679694925239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,243 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:25,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:25,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:25,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:25,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:25,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:25,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:25,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:25,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:25,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:25,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:25,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:25,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:25,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,274 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:25,274 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:25,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,375 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:25,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:25,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:25,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:25,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:25,379 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:25,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:25,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:25,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:25,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:25,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:25,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:25,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=772 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/192f4e259e0a417992fa6017118f8952 2023-03-24 21:54:25,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:25,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:25,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:25,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/1937bd20bb3943e29cde846362582a98 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:25,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:25,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:25,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:25,415 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:25,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:25,416 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98, entries=200, sequenceid=772, filesize=32.3 K 2023-03-24 21:54:25,417 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,418 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/950ba3d54aa149b2ac4a168f0410dab9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9 2023-03-24 21:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b2fb7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:25,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:25,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9, entries=150, sequenceid=772, filesize=11.9 K 2023-03-24 21:54:25,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/192f4e259e0a417992fa6017118f8952 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952 2023-03-24 21:54:25,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:25,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:25,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:25,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:25,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:25,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,431 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952, entries=150, sequenceid=772, filesize=11.9 K 2023-03-24 21:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:25,432 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:25,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:25,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:25,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:25,432 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 733bfd60f65777910098447a7a228a5c in 917ms, sequenceid=772, compaction requested=true 2023-03-24 21:54:25,432 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:25,432 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:25,432 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:25,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:25,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:25,433 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:25,433 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:25,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137544 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:25,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/A is initiating minor compaction (all files) 2023-03-24 21:54:25,435 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/A in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,435 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6d7bf067e9934999b1b8a9b19ff1de36, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=134.3 K 2023-03-24 21:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,435 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61204 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:25,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,435 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/B is initiating minor compaction (all files) 2023-03-24 21:54:25,435 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6d7bf067e9934999b1b8a9b19ff1de36, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859719 2023-03-24 21:54:25,436 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/B in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,436 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/16a51dcab9ec494494a5b96017b3791e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=59.8 K 2023-03-24 21:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:25,436 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 13fc2a2f248e4cb5a4da4a21a67cdcc4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679694859802 2023-03-24 21:54:25,436 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 16a51dcab9ec494494a5b96017b3791e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859719 2023-03-24 21:54:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,436 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8880541a3a3a40418e6fd21792e1b428, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679694859802 2023-03-24 21:54:25,437 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting bbbc745e1b20401fb407449cf54a0ced, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679694860941 2023-03-24 21:54:25,437 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 587ae11c2ba84052ab5b4528b5c873c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679694860941 2023-03-24 21:54:25,437 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8c8779469609491f8a04181839cc98ca, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1679694863077 2023-03-24 21:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,437 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c026c9f8deb34991904a7f6f0a065fff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1679694863077 2023-03-24 21:54:25,438 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1937bd20bb3943e29cde846362582a98, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:25,438 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 950ba3d54aa149b2ac4a168f0410dab9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:25,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:25,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:25,441 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:25,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:25,441 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:25,441 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:25,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:25,457 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#B#compaction#162 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:25,485 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#A#compaction#163 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:25,522 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/00bb50c0848841449855a047b3140ff3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/00bb50c0848841449855a047b3140ff3 2023-03-24 21:54:25,530 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/B of 733bfd60f65777910098447a7a228a5c into 00bb50c0848841449855a047b3140ff3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:25,530 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:25,530 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/B, priority=11, startTime=1679694865433; duration=0sec 2023-03-24 21:54:25,531 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:25,531 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:25,533 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61204 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:25,533 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/C is initiating minor compaction (all files) 2023-03-24 21:54:25,533 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/C in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,533 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/53d66b971c33448aa0d4a145b6a525c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=59.8 K 2023-03-24 21:54:25,534 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 53d66b971c33448aa0d4a145b6a525c4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694859719 2023-03-24 21:54:25,534 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 03d99e9e5a97489ca164bc5a824f386a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=709, earliestPutTs=1679694859802 2023-03-24 21:54:25,535 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e44bb730f6f04219a275027d63fdd9ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=723, earliestPutTs=1679694860941 2023-03-24 21:54:25,535 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e807c92cb1634a69a7b982209a6a7f1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=759, earliestPutTs=1679694863077 2023-03-24 21:54:25,536 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 192f4e259e0a417992fa6017118f8952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:25,541 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:25,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:25,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:25,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:25,550 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:25,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:25,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:25,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,553 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#C#compaction#164 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:25,553 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:25,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:25,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:25,554 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:25,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:25,563 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:25,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:25,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:25,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:25,584 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:25,584 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:25,585 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:25,585 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:25,639 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a00c0878771740fdb15405a9c4377f27 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a00c0878771740fdb15405a9c4377f27 2023-03-24 21:54:25,645 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=792, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/475dd57938a6473f9e2fc6999ba1cb01 2023-03-24 21:54:25,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:25,678 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:25,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694925715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,719 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694925715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,720 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694925715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694925820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694925820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:25,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694925820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:25,994 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d63464af10254f6391f23fea09e28869 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d63464af10254f6391f23fea09e28869 2023-03-24 21:54:25,998 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/C of 733bfd60f65777910098447a7a228a5c into d63464af10254f6391f23fea09e28869(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:25,998 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:25,999 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/C, priority=11, startTime=1679694865433; duration=0sec 2023-03-24 21:54:25,999 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:26,022 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694926022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694926023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694926024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,060 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/96313f55efe84b1fa6fa3a9c3e889448 2023-03-24 21:54:26,090 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/1626150f67d14777b2fef63b5fafb35f 2023-03-24 21:54:26,097 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/475dd57938a6473f9e2fc6999ba1cb01 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01 2023-03-24 21:54:26,102 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01, entries=150, sequenceid=792, filesize=25.4 K 2023-03-24 21:54:26,103 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/96313f55efe84b1fa6fa3a9c3e889448 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448 2023-03-24 21:54:26,111 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448, entries=150, sequenceid=792, filesize=11.9 K 2023-03-24 21:54:26,112 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/1626150f67d14777b2fef63b5fafb35f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f 2023-03-24 21:54:26,116 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f, entries=150, sequenceid=792, filesize=11.9 K 2023-03-24 21:54:26,117 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 733bfd60f65777910098447a7a228a5c in 533ms, sequenceid=792, compaction requested=false 2023-03-24 21:54:26,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:26,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:26,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:26,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:26,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:26,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:26,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,120 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:26,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,121 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:26,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:26,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fbeadf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:26,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:26,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:26,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:26,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:26,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:26,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:26,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:26,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:26,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:26,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:26,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:26,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:26,129 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:26,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:26,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:26,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:26,130 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:26,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:26,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:26,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:26,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:26,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:26,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:26,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:26,137 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,137 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:26,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:26,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:26,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:26,138 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:26,138 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:26,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:26,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:26,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:26,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:26,238 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:26,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:26,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:26,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:26,245 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:26,245 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:26,245 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:26,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:26,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:26,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:26,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:26,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:26,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:26,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:26,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:26,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:26,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:26,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:26,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:26,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:26,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:26,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:26,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:26,251 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:26,251 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:26,251 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:54:26,251 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:26,251 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:26,251 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:26,252 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:26,252 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:26,252 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:26,328 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:26,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:26,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694926347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694926348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,352 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694926351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,396 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/9b24e20f1a22421eb38e568d63ccc10a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/9b24e20f1a22421eb38e568d63ccc10a 2023-03-24 21:54:26,402 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/A of 733bfd60f65777910098447a7a228a5c into 9b24e20f1a22421eb38e568d63ccc10a(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:26,402 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:26,402 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/A, priority=11, startTime=1679694865432; duration=0sec 2023-03-24 21:54:26,402 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:26,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694926449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694926453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694926455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,655 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694926654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694926656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,657 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694926657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,690 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244fdb3cb8e3004204a77ad7150ba18c08 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244fdb3cb8e3004204a77ad7150ba18c08 2023-03-24 21:54:26,697 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=808, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/c3da735a29d64afaa99420884ff53fd1 2023-03-24 21:54:26,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694926958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,958 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694926958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:26,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:26,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694926958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,115 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/2920fe3367234ef3a0fc93b1769f38b1 2023-03-24 21:54:27,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:27,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694927234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,237 DEBUG [Thread-967] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:27,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:27,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58488 deadline: 1679694927244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,246 DEBUG [Thread-963] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:27,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:27,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694927460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:27,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694927460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,468 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:27,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694927463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,527 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/3f4c757ccd8f4b8ca0deac31a020f219 2023-03-24 21:54:27,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/c3da735a29d64afaa99420884ff53fd1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1 2023-03-24 21:54:27,536 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1, entries=150, sequenceid=808, filesize=25.4 K 2023-03-24 21:54:27,537 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/2920fe3367234ef3a0fc93b1769f38b1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1 2023-03-24 21:54:27,541 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1, entries=150, sequenceid=808, filesize=11.9 K 2023-03-24 21:54:27,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/3f4c757ccd8f4b8ca0deac31a020f219 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219 2023-03-24 21:54:27,546 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219, entries=150, sequenceid=808, filesize=11.9 K 2023-03-24 21:54:27,547 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 733bfd60f65777910098447a7a228a5c in 1296ms, sequenceid=808, compaction requested=true 2023-03-24 21:54:27,547 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:27,547 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:27,547 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:27,547 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:27,547 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,547 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:27,547 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:27,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:27,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:27,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:27,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:27,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:27,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e33a4b1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:27,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:27,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,552 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:27,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:27,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:27,552 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,553 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:27,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:27,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:27,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:27,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:27,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:27,561 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:27,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:27,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:27,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:27,561 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:27,562 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,562 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:27,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,562 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:27,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:27,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:27,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:27,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:27,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:27,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:27,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:27,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:27,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:27,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:27,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:27,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:27,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:27,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:27,567 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:27,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:27,567 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:27,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,667 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:27,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:27,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:27,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:27,671 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:27,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:27,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:27,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:27,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:27,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:27,672 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:27,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:27,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:27,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:27,674 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:27,674 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:27,674 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:27,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:27,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:27,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:27,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:27,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:27,674 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:27,674 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:27,675 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:27,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:27,688 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032433bcccdf3a1b4cc299f87df719edddf1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032433bcccdf3a1b4cc299f87df719edddf1 2023-03-24 21:54:28,096 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=827, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/f4a162c0452849e08e14bc3cc1716a9c 2023-03-24 21:54:28,117 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/e7da772f815646908b1c5b453c2fdf92 2023-03-24 21:54:28,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:28,465 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:28,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694928481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694928481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,486 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694928482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,534 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=827 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/84e63f97200b4e6f9631133b5cc49bc6 2023-03-24 21:54:28,543 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/f4a162c0452849e08e14bc3cc1716a9c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c 2023-03-24 21:54:28,548 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c, entries=150, sequenceid=827, filesize=25.4 K 2023-03-24 21:54:28,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/e7da772f815646908b1c5b453c2fdf92 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92 2023-03-24 21:54:28,560 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92, entries=150, sequenceid=827, filesize=11.9 K 2023-03-24 21:54:28,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/84e63f97200b4e6f9631133b5cc49bc6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6 2023-03-24 21:54:28,567 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6, entries=150, sequenceid=827, filesize=11.9 K 2023-03-24 21:54:28,568 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 733bfd60f65777910098447a7a228a5c in 893ms, sequenceid=827, compaction requested=true 2023-03-24 21:54:28,569 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:28,569 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:28,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:28,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:28,574 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:28,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fc9c2b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:28,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:28,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:28,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:28,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:28,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:28,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:28,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,584 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,585 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:28,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:28,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:28,585 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:28,585 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:28,586 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:28,587 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:28,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:28,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:28,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:28,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:28,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:28,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:28,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:28,596 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:28,596 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:28,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694928624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694928633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694928633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,696 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:28,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:28,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:28,700 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,705 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:28,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:28,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:28,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:28,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:28,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:28,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:28,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:28,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:28,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:28,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:28,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:28,707 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:28,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f350d14[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:28,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:28,723 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,724 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:28,724 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:28,724 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:28,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:28,726 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:28,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:28,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:28,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:28,728 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:28,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:28,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:28,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:28,729 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:28,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694928734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:28,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:28,736 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,736 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:28,737 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:28,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:28,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694928737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694928738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,837 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:28,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:28,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:28,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:28,841 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:28,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:28,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:28,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,844 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,844 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:28,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:28,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:28,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:28,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:28,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:28,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:28,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:28,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:28,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4502b93f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:28,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:28,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:28,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:28,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:28,857 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:28,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:28,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,858 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,858 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:28,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:28,860 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:28,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:28,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:28,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:28,861 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:28,862 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:28,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:28,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,868 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:28,868 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:28,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694928935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694928942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694928943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,969 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:28,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:28,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:28,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:28,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:28,972 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:28,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:28,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:28,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:28,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:28,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:28,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:28,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:28,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:28,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:28,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:28,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:28,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:28,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:28,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:28,976 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:28,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:28,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:28,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:28,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:28,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:28,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:28,978 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:28,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:28,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:28,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d79b02[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:28,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:28,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:28,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:28,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:28,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:28,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:28,983 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:28,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:28,989 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:28,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:28,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:28,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:28,989 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:28,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:28,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:28,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,001 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:29,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:29,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,001 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:29,001 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:29,025 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303241ff7c81913b949b98d04c959849cad06 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241ff7c81913b949b98d04c959849cad06 2023-03-24 21:54:29,031 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=843, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/2ba772b626e84866867fd322e115c90e 2023-03-24 21:54:29,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/673572d71e6c4399bc36f979da92d49a 2023-03-24 21:54:29,102 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:29,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:29,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:29,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:29,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:29,106 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:29,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:29,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:29,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:29,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:29,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:29,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:29,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:29,111 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:29,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62ae4237[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:29,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:29,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:29,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:29,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:29,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:29,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:29,121 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:29,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:29,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:29,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:29,121 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:29,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,123 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:29,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,128 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:29,128 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:29,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,228 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:29,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:29,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:29,233 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:29,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,236 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:29,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:29,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:29,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694929238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:29,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:29,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:29,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:29,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:29,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:29,243 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,247 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694929247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694929247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:29,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1504bedb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:29,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:29,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:29,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:29,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:29,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:29,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:29,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:29,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:29,255 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:29,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:29,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:29,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:29,255 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:29,258 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,258 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,258 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:29,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:29,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,266 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:29,266 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:29,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,366 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:29,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:29,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:29,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:29,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:29,375 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:29,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:29,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:29,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,377 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,378 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:29,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:29,379 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:29,379 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:29,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:29,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:29,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:29,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:29,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:29,397 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,397 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,397 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,397 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:29,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@451f2605[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:29,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:29,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:29,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:29,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:29,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:29,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:29,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:29,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:29,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:29,414 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:29,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:29,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:29,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:29,414 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:29,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:29,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:29,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:29,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:29,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:29,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:29,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:29,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:29,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:29,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:29,429 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,429 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:29,429 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:29,452 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=843 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/6d0aec238fbb4b2a9667db44fce10531 2023-03-24 21:54:29,457 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/2ba772b626e84866867fd322e115c90e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e 2023-03-24 21:54:29,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e, entries=200, sequenceid=843, filesize=32.3 K 2023-03-24 21:54:29,462 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/673572d71e6c4399bc36f979da92d49a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a 2023-03-24 21:54:29,466 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 21:54:29,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/6d0aec238fbb4b2a9667db44fce10531 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531 2023-03-24 21:54:29,470 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531, entries=150, sequenceid=843, filesize=11.9 K 2023-03-24 21:54:29,471 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 733bfd60f65777910098447a7a228a5c in 886ms, sequenceid=843, compaction requested=true 2023-03-24 21:54:29,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:29,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:29,472 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:29,472 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:29,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:29,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:29,474 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 137714 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:29,474 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/A is initiating minor compaction (all files) 2023-03-24 21:54:29,474 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/A in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,474 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/9b24e20f1a22421eb38e568d63ccc10a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=134.5 K 2023-03-24 21:54:29,474 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:29,474 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/B is initiating minor compaction (all files) 2023-03-24 21:54:29,474 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/B in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,474 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/00bb50c0848841449855a047b3140ff3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=59.9 K 2023-03-24 21:54:29,475 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9b24e20f1a22421eb38e568d63ccc10a, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:29,475 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 00bb50c0848841449855a047b3140ff3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:29,484 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 475dd57938a6473f9e2fc6999ba1cb01, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694864539 2023-03-24 21:54:29,484 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 96313f55efe84b1fa6fa3a9c3e889448, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694864539 2023-03-24 21:54:29,484 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c3da735a29d64afaa99420884ff53fd1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694865681 2023-03-24 21:54:29,484 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2920fe3367234ef3a0fc93b1769f38b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694865681 2023-03-24 21:54:29,484 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f4a162c0452849e08e14bc3cc1716a9c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694866340 2023-03-24 21:54:29,485 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e7da772f815646908b1c5b453c2fdf92, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694866340 2023-03-24 21:54:29,485 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2ba772b626e84866867fd322e115c90e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868476 2023-03-24 21:54:29,485 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 673572d71e6c4399bc36f979da92d49a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868480 2023-03-24 21:54:29,496 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#B#compaction#178 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:29,504 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#A#compaction#177 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:29,529 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:29,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:29,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:29,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:29,540 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:29,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:29,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:29,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,543 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:29,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:29,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:29,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:29,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:29,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:29,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:29,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:29,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:29,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:29,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:29,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:29,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:29,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:29,551 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:29,552 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:29,599 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032491d88f7b10364474a2ae283ca28ea59c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032491d88f7b10364474a2ae283ca28ea59c 2023-03-24 21:54:29,627 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=859, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/4bee0fef8c4f477c817a70251135a492 2023-03-24 21:54:29,656 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/50c0ff416b024d5e84a90d7cbc083edf 2023-03-24 21:54:29,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:29,749 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:29,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694929776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694929777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694929777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694929878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694929882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:29,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694929882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:29,925 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/2064a198d0c54aaebff11b5ac7ea6407 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2064a198d0c54aaebff11b5ac7ea6407 2023-03-24 21:54:29,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/B of 733bfd60f65777910098447a7a228a5c into 2064a198d0c54aaebff11b5ac7ea6407(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:29,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/B, priority=11, startTime=1679694869472; duration=0sec 2023-03-24 21:54:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:54:29,932 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:54:29,932 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/C is initiating minor compaction (all files) 2023-03-24 21:54:29,932 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/C in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:29,932 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d63464af10254f6391f23fea09e28869, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=59.9 K 2023-03-24 21:54:29,933 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d63464af10254f6391f23fea09e28869, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=772, earliestPutTs=1679694864202 2023-03-24 21:54:29,933 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1626150f67d14777b2fef63b5fafb35f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694864539 2023-03-24 21:54:29,934 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3f4c757ccd8f4b8ca0deac31a020f219, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694865681 2023-03-24 21:54:29,935 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 84e63f97200b4e6f9631133b5cc49bc6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=827, earliestPutTs=1679694866340 2023-03-24 21:54:29,935 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6d0aec238fbb4b2a9667db44fce10531, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868480 2023-03-24 21:54:29,948 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#C#compaction#182 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:29,960 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/703d1dc5cf3e400397a3146959eb2095 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/703d1dc5cf3e400397a3146959eb2095 2023-03-24 21:54:29,965 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/A of 733bfd60f65777910098447a7a228a5c into 703d1dc5cf3e400397a3146959eb2095(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:29,965 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:29,965 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/A, priority=11, startTime=1679694869472; duration=0sec 2023-03-24 21:54:29,965 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:30,077 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=859 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/151cc15cf5774f3796135f63887a8583 2023-03-24 21:54:30,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/4bee0fef8c4f477c817a70251135a492 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492 2023-03-24 21:54:30,086 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694930083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,088 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694930088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,090 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694930090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,091 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492, entries=100, sequenceid=859, filesize=18.6 K 2023-03-24 21:54:30,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/50c0ff416b024d5e84a90d7cbc083edf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf 2023-03-24 21:54:30,099 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf, entries=100, sequenceid=859, filesize=9.5 K 2023-03-24 21:54:30,102 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/151cc15cf5774f3796135f63887a8583 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583 2023-03-24 21:54:30,114 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583, entries=100, sequenceid=859, filesize=9.5 K 2023-03-24 21:54:30,115 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 733bfd60f65777910098447a7a228a5c in 564ms, sequenceid=859, compaction requested=false 2023-03-24 21:54:30,115 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:30,115 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:30,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:30,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:30,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:30,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:30,118 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,118 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:30,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:30,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7faaaa2f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:30,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:30,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:30,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:30,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:30,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:30,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,125 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:30,125 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,125 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:30,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:30,128 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:30,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:30,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:30,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:30,128 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:30,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:30,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:30,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:30,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:30,135 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:30,136 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:30,135 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:30,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:30,136 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,136 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,236 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:30,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:30,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:30,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:30,239 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:30,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:30,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:30,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:30,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,241 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:30,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:30,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:30,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:30,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:30,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:30,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:30,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:30,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:30,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:30,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:30,243 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:30,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,259 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b2bf1a60552141e0b295b25fe70908cb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b2bf1a60552141e0b295b25fe70908cb 2023-03-24 21:54:30,265 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=878, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/8aaa5d45ec824b1786dd0dc7b19d2754 2023-03-24 21:54:30,294 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/4557a9d7a16841619d561aa0fc7b364a 2023-03-24 21:54:30,360 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/d925a043c9a04fa2b53cfe384e28ed58 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d925a043c9a04fa2b53cfe384e28ed58 2023-03-24 21:54:30,365 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 733bfd60f65777910098447a7a228a5c/C of 733bfd60f65777910098447a7a228a5c into d925a043c9a04fa2b53cfe384e28ed58(size=12.5 K), total size for store is 22.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:30,365 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:30,365 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/C, priority=11, startTime=1679694869472; duration=0sec 2023-03-24 21:54:30,365 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:30,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:30,389 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694930428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694930428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,435 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694930428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694930537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694930537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694930538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,709 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=878 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/b2999e4978a24d3495a797384541633f 2023-03-24 21:54:30,714 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/8aaa5d45ec824b1786dd0dc7b19d2754 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754 2023-03-24 21:54:30,718 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754, entries=150, sequenceid=878, filesize=25.4 K 2023-03-24 21:54:30,719 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/4557a9d7a16841619d561aa0fc7b364a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a 2023-03-24 21:54:30,725 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a, entries=150, sequenceid=878, filesize=11.9 K 2023-03-24 21:54:30,728 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/b2999e4978a24d3495a797384541633f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f 2023-03-24 21:54:30,731 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f, entries=150, sequenceid=878, filesize=11.9 K 2023-03-24 21:54:30,732 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 733bfd60f65777910098447a7a228a5c in 489ms, sequenceid=878, compaction requested=true 2023-03-24 21:54:30,732 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:30,732 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:30,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:30,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:30,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:30,732 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:30,735 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:30,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f05f017[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:30,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:30,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,738 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:30,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:30,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:30,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:30,740 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:30,740 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,740 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,740 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:30,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:30,743 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:30,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:30,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:30,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:30,743 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:30,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:30,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:30,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:30,746 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:54:30,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:30,747 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:30,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,747 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:30,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:30,753 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,753 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,753 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:30,755 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:30,755 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:30,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694930761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694930762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694930763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,855 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:30,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:30,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:30,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:30,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:30,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:30,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:30,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:30,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,862 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:30,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:30,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,863 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:30,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:30,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:30,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:30,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:30,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:30,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,864 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:30,866 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:30,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:30,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694930863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,866 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:30,866 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:30,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:30,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:30,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694930866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,868 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694930866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,870 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:30,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:30,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f47794a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:30,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,873 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:30,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:30,873 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:30,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:30,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:30,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:30,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:30,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:30,879 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:30,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:30,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:30,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:30,879 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:30,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:30,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:30,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:30,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:30,887 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:30,887 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,887 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:30,888 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:30,888 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,888 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:30,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,888 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:30,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:30,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,988 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:30,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:30,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:30,993 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:30,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:30,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:30,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:30,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:30,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:30,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:30,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:30,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:30,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,000 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,001 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,001 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,001 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,001 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,003 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d453a9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,008 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,008 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,017 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,017 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,029 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,029 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,071 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694931068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694931070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694931070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,129 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,135 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,140 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,142 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,142 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,142 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,143 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,144 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4985662e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,151 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,153 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,153 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,160 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,161 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,161 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,160 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:31,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,169 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a4d11765463d446ea91fa71c308a959b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a4d11765463d446ea91fa71c308a959b 2023-03-24 21:54:31,184 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=897, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/32f4a3d564de4f30ae7fc8df985bd37e 2023-03-24 21:54:31,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58508 deadline: 1679694931244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,249 DEBUG [Thread-967] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:31,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58488 deadline: 1679694931252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,257 DEBUG [Thread-963] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:31,261 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,264 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,264 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,264 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,264 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,271 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,272 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,272 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,272 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,272 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9c0a241[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,278 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,284 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,287 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:31,287 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694931374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694931374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694931376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,387 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,390 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,392 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,393 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,394 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,394 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,394 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,395 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,395 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,396 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,397 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,397 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,398 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d29171a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,398 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,409 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,410 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,413 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,415 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,415 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,415 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,515 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,519 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,522 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,522 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,522 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,523 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,523 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,524 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,525 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c21580d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,530 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,532 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,532 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,536 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:31,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,538 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,538 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/f8f8925aefc44263ac99498b9ce17c4c 2023-03-24 21:54:31,639 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,673 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,673 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ef7efb1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,676 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,676 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,682 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,683 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,686 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,686 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,693 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,693 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,794 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,797 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,798 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,798 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,800 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,802 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,802 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,802 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,802 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,802 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,802 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,802 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,802 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,802 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,802 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,802 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,802 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,804 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,805 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,805 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42a98e84[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,807 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,814 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,814 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,815 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,815 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,815 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,816 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,820 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:31,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,820 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58468 deadline: 1679694931877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58490 deadline: 1679694931879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:31,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:58502 deadline: 1679694931880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,920 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:31,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:31,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:31,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:31,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:31,923 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:31,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:31,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:31,924 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,924 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,926 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,926 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:31,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,926 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:31,927 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:31,927 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:31,927 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,927 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:31,927 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:31,927 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:31,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:31,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:31,930 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:31,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@679dd00c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:31,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:31,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,933 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:31,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:31,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:31,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:31,934 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:31,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:31,943 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:31,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:31,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:31,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:31,943 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:31,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:31,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:31,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:31,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,956 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,956 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:31,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:31,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:31,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:31,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:31,957 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:31,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:31,957 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:32,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=897 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/a3bab7ea01fd4126858278fc8c5e242d 2023-03-24 21:54:32,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/32f4a3d564de4f30ae7fc8df985bd37e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e 2023-03-24 21:54:32,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e, entries=250, sequenceid=897, filesize=39.2 K 2023-03-24 21:54:32,032 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/f8f8925aefc44263ac99498b9ce17c4c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c 2023-03-24 21:54:32,036 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c, entries=150, sequenceid=897, filesize=11.9 K 2023-03-24 21:54:32,037 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/a3bab7ea01fd4126858278fc8c5e242d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d 2023-03-24 21:54:32,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d, entries=150, sequenceid=897, filesize=11.9 K 2023-03-24 21:54:32,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 733bfd60f65777910098447a7a228a5c in 1297ms, sequenceid=897, compaction requested=true 2023-03-24 21:54:32,043 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:32,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:32,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:54:32,043 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 21:54:32,043 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:32,044 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:32,045 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:32,045 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/A is initiating minor compaction (all files) 2023-03-24 21:54:32,045 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/A in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:32,045 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/703d1dc5cf3e400397a3146959eb2095, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=109.2 K 2023-03-24 21:54:32,045 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:32,045 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/C is initiating minor compaction (all files) 2023-03-24 21:54:32,045 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/C in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:32,045 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 703d1dc5cf3e400397a3146959eb2095, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868480 2023-03-24 21:54:32,045 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d925a043c9a04fa2b53cfe384e28ed58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=45.8 K 2023-03-24 21:54:32,046 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4bee0fef8c4f477c817a70251135a492, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1679694868605 2023-03-24 21:54:32,046 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d925a043c9a04fa2b53cfe384e28ed58, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868480 2023-03-24 21:54:32,046 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8aaa5d45ec824b1786dd0dc7b19d2754, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=878, earliestPutTs=1679694869761 2023-03-24 21:54:32,047 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 32f4a3d564de4f30ae7fc8df985bd37e, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1679694870396 2023-03-24 21:54:32,047 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 151cc15cf5774f3796135f63887a8583, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1679694868605 2023-03-24 21:54:32,047 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b2999e4978a24d3495a797384541633f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=878, earliestPutTs=1679694869761 2023-03-24 21:54:32,048 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a3bab7ea01fd4126858278fc8c5e242d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1679694870400 2023-03-24 21:54:32,057 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:32,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:32,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:32,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:32,063 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:32,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:32,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:32,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:32,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:32,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:32,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:32,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:32,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:32,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:32,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:32,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:32,073 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:32,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:32,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:32,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:32,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:32,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:32,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:32,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:32,088 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. started... 2023-03-24 21:54:32,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:32,089 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:32,089 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:32,091 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#A#compaction#189 average throughput is 0.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:32,103 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/bea354d18ea64257af3d6a17ccd47676 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/bea354d18ea64257af3d6a17ccd47676 2023-03-24 21:54:32,110 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 733bfd60f65777910098447a7a228a5c/C of 733bfd60f65777910098447a7a228a5c into bea354d18ea64257af3d6a17ccd47676(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:32,111 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:32,111 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/C, priority=12, startTime=1679694872043; duration=0sec 2023-03-24 21:54:32,111 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:32,111 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:32,112 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e9488e20d9234fc2add966e9888fb407 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9488e20d9234fc2add966e9888fb407 2023-03-24 21:54:32,113 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:32,113 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 733bfd60f65777910098447a7a228a5c/B is initiating minor compaction (all files) 2023-03-24 21:54:32,113 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 733bfd60f65777910098447a7a228a5c/B in TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:32,113 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2064a198d0c54aaebff11b5ac7ea6407, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp, totalSize=45.8 K 2023-03-24 21:54:32,115 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2064a198d0c54aaebff11b5ac7ea6407, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=843, earliestPutTs=1679694868480 2023-03-24 21:54:32,115 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 50c0ff416b024d5e84a90d7cbc083edf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=859, earliestPutTs=1679694868605 2023-03-24 21:54:32,116 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4557a9d7a16841619d561aa0fc7b364a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=878, earliestPutTs=1679694869761 2023-03-24 21:54:32,117 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f8f8925aefc44263ac99498b9ce17c4c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=897, earliestPutTs=1679694870400 2023-03-24 21:54:32,127 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 733bfd60f65777910098447a7a228a5c#B#compaction#192 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:32,129 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=911, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/95d84b16aaa14a0da9add4c2a934e017 2023-03-24 21:54:32,531 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/ad04f59678764da9952176a74f1387de as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ad04f59678764da9952176a74f1387de 2023-03-24 21:54:32,535 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 733bfd60f65777910098447a7a228a5c/A of 733bfd60f65777910098447a7a228a5c into ad04f59678764da9952176a74f1387de(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:32,535 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:32,535 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/A, priority=12, startTime=1679694872043; duration=0sec 2023-03-24 21:54:32,535 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:32,540 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/dbf95336947d47619ebaea3b3c8a4ba6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/dbf95336947d47619ebaea3b3c8a4ba6 2023-03-24 21:54:32,544 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 733bfd60f65777910098447a7a228a5c/B of 733bfd60f65777910098447a7a228a5c into dbf95336947d47619ebaea3b3c8a4ba6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:32,544 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:32,544 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c., storeName=733bfd60f65777910098447a7a228a5c/B, priority=12, startTime=1679694872043; duration=0sec 2023-03-24 21:54:32,544 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:32,550 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/65ca9973e7bf415cb3c31b79d6ac00f4 2023-03-24 21:54:32,803 DEBUG [Thread-972] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x16549bf9 to 127.0.0.1:57724 2023-03-24 21:54:32,803 DEBUG [Thread-972] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,804 DEBUG [Thread-974] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x462bd7ef to 127.0.0.1:57724 2023-03-24 21:54:32,804 DEBUG [Thread-974] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,804 DEBUG [Thread-978] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50f79162 to 127.0.0.1:57724 2023-03-24 21:54:32,804 DEBUG [Thread-978] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,805 DEBUG [Thread-976] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1529328e to 127.0.0.1:57724 2023-03-24 21:54:32,805 DEBUG [Thread-976] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:32,888 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. as already flushing 2023-03-24 21:54:32,888 DEBUG [Thread-969] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a286c6d to 127.0.0.1:57724 2023-03-24 21:54:32,888 DEBUG [Thread-969] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,890 DEBUG [Thread-961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ed04176 to 127.0.0.1:57724 2023-03-24 21:54:32,890 DEBUG [Thread-961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,891 DEBUG [Thread-965] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2a2c2ab1 to 127.0.0.1:57724 2023-03-24 21:54:32,891 DEBUG [Thread-965] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:32,962 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=911 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/f50171d1f8cb43518d74d164a8697b3b 2023-03-24 21:54:32,967 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/95d84b16aaa14a0da9add4c2a934e017 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/95d84b16aaa14a0da9add4c2a934e017 2023-03-24 21:54:32,971 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/95d84b16aaa14a0da9add4c2a934e017, entries=150, sequenceid=911, filesize=25.4 K 2023-03-24 21:54:32,972 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/65ca9973e7bf415cb3c31b79d6ac00f4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/65ca9973e7bf415cb3c31b79d6ac00f4 2023-03-24 21:54:32,975 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/65ca9973e7bf415cb3c31b79d6ac00f4, entries=150, sequenceid=911, filesize=11.9 K 2023-03-24 21:54:32,976 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/f50171d1f8cb43518d74d164a8697b3b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/f50171d1f8cb43518d74d164a8697b3b 2023-03-24 21:54:32,980 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/f50171d1f8cb43518d74d164a8697b3b, entries=150, sequenceid=911, filesize=11.9 K 2023-03-24 21:54:32,981 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=20.13 KB/20610 for 733bfd60f65777910098447a7a228a5c in 893ms, sequenceid=911, compaction requested=false 2023-03-24 21:54:32,981 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:32,981 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:32,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:32,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:32,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:32,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:32,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:32,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:32,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41da1dc2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:32,986 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:32,986 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:32,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:32,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:32,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:32,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:32,988 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:32,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:32,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:32,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:32,991 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:32,991 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:32,991 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:32,991 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:32,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:32,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:32,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:32,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:32,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:32,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:32,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:32,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:32,997 DEBUG [Thread-971] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:32,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:32,997 DEBUG [Thread-971] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:32,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:33,097 DEBUG [Thread-971] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:33,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:41,051 DEBUG [regionserver/jenkins-hbase9:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 21:54:41,282 DEBUG [Thread-967] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x642eab2b to 127.0.0.1:57724 2023-03-24 21:54:41,282 DEBUG [Thread-967] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:41,291 DEBUG [Thread-963] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41982377 to 127.0.0.1:57724 2023-03-24 21:54:41,291 DEBUG [Thread-963] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 76 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 130 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 373 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 160 2023-03-24 21:54:41,291 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4837 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4862 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 2057 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 6170 rows 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 2037 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 6111 rows 2023-03-24 21:54:41,292 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:54:41,292 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5661dd0b to 127.0.0.1:57724 2023-03-24 21:54:41,292 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:54:41,295 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:54:41,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:54:41,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:41,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:41,300 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694881300"}]},"ts":"1679694881300"} 2023-03-24 21:54:41,301 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:54:41,302 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:54:41,302 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, UNASSIGN}] 2023-03-24 21:54:41,304 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, UNASSIGN 2023-03-24 21:54:41,304 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:41,304 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694881304"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694881304"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694881304"}]},"ts":"1679694881304"} 2023-03-24 21:54:41,305 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:41,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:41,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 733bfd60f65777910098447a7a228a5c, disabling compactions & flushes 2023-03-24 21:54:41,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. after waiting 0 ms 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:41,457 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 733bfd60f65777910098447a7a228a5c 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=A 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=B 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 733bfd60f65777910098447a7a228a5c, store=C 2023-03-24 21:54:41,457 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:41,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:41,872 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249c908c8299504e0596adf468f4998484 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c908c8299504e0596adf468f4998484 2023-03-24 21:54:41,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:42,277 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=922, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/cdf8cd3c757842109d361ac1bcfa098b 2023-03-24 21:54:42,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:42,686 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=922 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/cfee3f2e26f24cc3bb534c0ec52611f0 2023-03-24 21:54:43,095 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=922 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/0b36f4429e3645baad25b5f65bbbc5f4 2023-03-24 21:54:43,099 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/A/cdf8cd3c757842109d361ac1bcfa098b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cdf8cd3c757842109d361ac1bcfa098b 2023-03-24 21:54:43,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cdf8cd3c757842109d361ac1bcfa098b, entries=150, sequenceid=922, filesize=25.4 K 2023-03-24 21:54:43,103 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/B/cfee3f2e26f24cc3bb534c0ec52611f0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/cfee3f2e26f24cc3bb534c0ec52611f0 2023-03-24 21:54:43,106 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/cfee3f2e26f24cc3bb534c0ec52611f0, entries=150, sequenceid=922, filesize=11.9 K 2023-03-24 21:54:43,107 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/.tmp/C/0b36f4429e3645baad25b5f65bbbc5f4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/0b36f4429e3645baad25b5f65bbbc5f4 2023-03-24 21:54:43,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/0b36f4429e3645baad25b5f65bbbc5f4, entries=150, sequenceid=922, filesize=11.9 K 2023-03-24 21:54:43,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 733bfd60f65777910098447a7a228a5c in 1654ms, sequenceid=922, compaction requested=true 2023-03-24 21:54:43,130 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cf060e620701401f950bba1e6a9b1c45, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6d7bf067e9934999b1b8a9b19ff1de36, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/9b24e20f1a22421eb38e568d63ccc10a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/703d1dc5cf3e400397a3146959eb2095, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e] to archive 2023-03-24 21:54:43,131 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:43,132 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c65687e07be0415e80523835abb08713 2023-03-24 21:54:43,133 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6ae5b02063294f2a87fd857833aa0a3a 2023-03-24 21:54:43,134 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/a38e3c9a918c4ce48bb5f570802bfa40 2023-03-24 21:54:43,135 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cf060e620701401f950bba1e6a9b1c45 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cf060e620701401f950bba1e6a9b1c45 2023-03-24 21:54:43,137 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1f53daf356d0450390f18d2cdd9681ed 2023-03-24 21:54:43,138 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6b65f54176124ad3a10dcc3c7fbdaadf 2023-03-24 21:54:43,139 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ddb97f82e25a4ef5a50cd0321c8ef9f7 2023-03-24 21:54:43,140 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/7703bdee9d52400ebb345e07a8111692 2023-03-24 21:54:43,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6d7bf067e9934999b1b8a9b19ff1de36 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/6d7bf067e9934999b1b8a9b19ff1de36 2023-03-24 21:54:43,141 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/13fc2a2f248e4cb5a4da4a21a67cdcc4 2023-03-24 21:54:43,142 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/bbbc745e1b20401fb407449cf54a0ced 2023-03-24 21:54:43,144 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8c8779469609491f8a04181839cc98ca 2023-03-24 21:54:43,145 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/1937bd20bb3943e29cde846362582a98 2023-03-24 21:54:43,146 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/9b24e20f1a22421eb38e568d63ccc10a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/9b24e20f1a22421eb38e568d63ccc10a 2023-03-24 21:54:43,147 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/475dd57938a6473f9e2fc6999ba1cb01 2023-03-24 21:54:43,148 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/c3da735a29d64afaa99420884ff53fd1 2023-03-24 21:54:43,149 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/f4a162c0452849e08e14bc3cc1716a9c 2023-03-24 21:54:43,150 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/2ba772b626e84866867fd322e115c90e 2023-03-24 21:54:43,151 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/703d1dc5cf3e400397a3146959eb2095 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/703d1dc5cf3e400397a3146959eb2095 2023-03-24 21:54:43,152 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/4bee0fef8c4f477c817a70251135a492 2023-03-24 21:54:43,153 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/8aaa5d45ec824b1786dd0dc7b19d2754 2023-03-24 21:54:43,154 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/32f4a3d564de4f30ae7fc8df985bd37e 2023-03-24 21:54:43,156 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8de558ff75a24776b7623de03dedc66f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/16a51dcab9ec494494a5b96017b3791e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/00bb50c0848841449855a047b3140ff3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2064a198d0c54aaebff11b5ac7ea6407, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c] to archive 2023-03-24 21:54:43,157 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:43,158 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/79333fe6505e4ebcb0e7a54a00421177 2023-03-24 21:54:43,160 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c9b0be64278a4da2a55e0bf9bda0c3f3 2023-03-24 21:54:43,161 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8de558ff75a24776b7623de03dedc66f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8de558ff75a24776b7623de03dedc66f 2023-03-24 21:54:43,162 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/3fbd6ec82784489286c5918dd27153c3 2023-03-24 21:54:43,163 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/bd0969d733164b199103550931ba0787 2023-03-24 21:54:43,164 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/ee8eee78549446f2a1a06a1fa2d1c910 2023-03-24 21:54:43,165 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/9a24a4906ead4e8ab1f0dc62d6e03cee 2023-03-24 21:54:43,166 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/16a51dcab9ec494494a5b96017b3791e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/16a51dcab9ec494494a5b96017b3791e 2023-03-24 21:54:43,167 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/98603c55d5614f568340b95af5badd1c 2023-03-24 21:54:43,168 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/8880541a3a3a40418e6fd21792e1b428 2023-03-24 21:54:43,169 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/587ae11c2ba84052ab5b4528b5c873c6 2023-03-24 21:54:43,170 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/c026c9f8deb34991904a7f6f0a065fff 2023-03-24 21:54:43,171 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/00bb50c0848841449855a047b3140ff3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/00bb50c0848841449855a047b3140ff3 2023-03-24 21:54:43,172 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/950ba3d54aa149b2ac4a168f0410dab9 2023-03-24 21:54:43,173 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/96313f55efe84b1fa6fa3a9c3e889448 2023-03-24 21:54:43,174 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2920fe3367234ef3a0fc93b1769f38b1 2023-03-24 21:54:43,175 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/e7da772f815646908b1c5b453c2fdf92 2023-03-24 21:54:43,176 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2064a198d0c54aaebff11b5ac7ea6407 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/2064a198d0c54aaebff11b5ac7ea6407 2023-03-24 21:54:43,177 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/673572d71e6c4399bc36f979da92d49a 2023-03-24 21:54:43,178 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/50c0ff416b024d5e84a90d7cbc083edf 2023-03-24 21:54:43,179 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/4557a9d7a16841619d561aa0fc7b364a 2023-03-24 21:54:43,180 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/f8f8925aefc44263ac99498b9ce17c4c 2023-03-24 21:54:43,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/7ef5b61d24574355a444c03b56b3ded6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/53d66b971c33448aa0d4a145b6a525c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d63464af10254f6391f23fea09e28869, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d925a043c9a04fa2b53cfe384e28ed58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d] to archive 2023-03-24 21:54:43,183 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:54:43,185 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/708e1e7d123b429d8484fcd1a62119bf 2023-03-24 21:54:43,186 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/fdbe92db0ed04ba1add9c9f5c26e866b 2023-03-24 21:54:43,187 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/7ef5b61d24574355a444c03b56b3ded6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/7ef5b61d24574355a444c03b56b3ded6 2023-03-24 21:54:43,188 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5506e378a0e4176abad0f1c548af160 2023-03-24 21:54:43,189 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/cba21da84eba40cba146afcfa016e47e 2023-03-24 21:54:43,190 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/17d4f2881ac642818a432dbe67237a94 2023-03-24 21:54:43,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/718f526a818e4fc99f907d36942fc88b 2023-03-24 21:54:43,191 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/53d66b971c33448aa0d4a145b6a525c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/53d66b971c33448aa0d4a145b6a525c4 2023-03-24 21:54:43,192 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d5f21bf1dae14e19879519f7b8e932a7 2023-03-24 21:54:43,193 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/03d99e9e5a97489ca164bc5a824f386a 2023-03-24 21:54:43,194 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e44bb730f6f04219a275027d63fdd9ca 2023-03-24 21:54:43,195 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/e807c92cb1634a69a7b982209a6a7f1a 2023-03-24 21:54:43,196 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d63464af10254f6391f23fea09e28869 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d63464af10254f6391f23fea09e28869 2023-03-24 21:54:43,197 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/192f4e259e0a417992fa6017118f8952 2023-03-24 21:54:43,198 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/1626150f67d14777b2fef63b5fafb35f 2023-03-24 21:54:43,199 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/3f4c757ccd8f4b8ca0deac31a020f219 2023-03-24 21:54:43,200 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/84e63f97200b4e6f9631133b5cc49bc6 2023-03-24 21:54:43,201 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d925a043c9a04fa2b53cfe384e28ed58 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/d925a043c9a04fa2b53cfe384e28ed58 2023-03-24 21:54:43,202 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/6d0aec238fbb4b2a9667db44fce10531 2023-03-24 21:54:43,203 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/151cc15cf5774f3796135f63887a8583 2023-03-24 21:54:43,204 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/b2999e4978a24d3495a797384541633f 2023-03-24 21:54:43,205 DEBUG [StoreCloser-TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/a3bab7ea01fd4126858278fc8c5e242d 2023-03-24 21:54:43,209 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits/925.seqid, newMaxSeqId=925, maxSeqId=4 2023-03-24 21:54:43,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c. 2023-03-24 21:54:43,211 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 733bfd60f65777910098447a7a228a5c: 2023-03-24 21:54:43,213 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:43,213 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=733bfd60f65777910098447a7a228a5c, regionState=CLOSED 2023-03-24 21:54:43,213 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694883213"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694883213"}]},"ts":"1679694883213"} 2023-03-24 21:54:43,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-24 21:54:43,215 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 733bfd60f65777910098447a7a228a5c, server=jenkins-hbase9.apache.org,45673,1679694819052 in 1.9090 sec 2023-03-24 21:54:43,216 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-24 21:54:43,217 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=733bfd60f65777910098447a7a228a5c, UNASSIGN in 1.9130 sec 2023-03-24 21:54:43,217 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694883217"}]},"ts":"1679694883217"} 2023-03-24 21:54:43,218 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:54:43,219 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:54:43,220 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9240 sec 2023-03-24 21:54:43,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-24 21:54:43,403 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-24 21:54:43,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:54:43,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,406 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 21:54:43,406 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,411 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:43,413 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits] 2023-03-24 21:54:43,415 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/95d84b16aaa14a0da9add4c2a934e017 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/95d84b16aaa14a0da9add4c2a934e017 2023-03-24 21:54:43,417 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ad04f59678764da9952176a74f1387de to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/ad04f59678764da9952176a74f1387de 2023-03-24 21:54:43,418 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cdf8cd3c757842109d361ac1bcfa098b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/A/cdf8cd3c757842109d361ac1bcfa098b 2023-03-24 21:54:43,420 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/65ca9973e7bf415cb3c31b79d6ac00f4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/65ca9973e7bf415cb3c31b79d6ac00f4 2023-03-24 21:54:43,421 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/cfee3f2e26f24cc3bb534c0ec52611f0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/cfee3f2e26f24cc3bb534c0ec52611f0 2023-03-24 21:54:43,423 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/dbf95336947d47619ebaea3b3c8a4ba6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/B/dbf95336947d47619ebaea3b3c8a4ba6 2023-03-24 21:54:43,425 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/0b36f4429e3645baad25b5f65bbbc5f4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/0b36f4429e3645baad25b5f65bbbc5f4 2023-03-24 21:54:43,426 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/bea354d18ea64257af3d6a17ccd47676 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/bea354d18ea64257af3d6a17ccd47676 2023-03-24 21:54:43,427 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/f50171d1f8cb43518d74d164a8697b3b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/C/f50171d1f8cb43518d74d164a8697b3b 2023-03-24 21:54:43,429 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits/925.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c/recovered.edits/925.seqid 2023-03-24 21:54:43,430 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/733bfd60f65777910098447a7a228a5c 2023-03-24 21:54:43,430 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:54:43,430 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:54:43,431 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 21:54:43,435 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032409c414926d58465983d5a20343fa8ec3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032409c414926d58465983d5a20343fa8ec3 2023-03-24 21:54:43,437 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241694fe1c428143ae923817c0c9ee9bee to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241694fe1c428143ae923817c0c9ee9bee 2023-03-24 21:54:43,439 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241f863ab65ea04c5089a9792de37a2191 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241f863ab65ea04c5089a9792de37a2191 2023-03-24 21:54:43,440 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241ff7c81913b949b98d04c959849cad06 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303241ff7c81913b949b98d04c959849cad06 2023-03-24 21:54:43,441 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242b1bc00c88314d029bf389531207e593 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242b1bc00c88314d029bf389531207e593 2023-03-24 21:54:43,443 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032433bcccdf3a1b4cc299f87df719edddf1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032433bcccdf3a1b4cc299f87df719edddf1 2023-03-24 21:54:43,444 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032440faf0a979054ce3ae8c4eb3af09150a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032440faf0a979054ce3ae8c4eb3af09150a 2023-03-24 21:54:43,446 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244fdb3cb8e3004204a77ad7150ba18c08 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244fdb3cb8e3004204a77ad7150ba18c08 2023-03-24 21:54:43,448 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032458838345b7794f0899c8d159ed58407a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032458838345b7794f0899c8d159ed58407a 2023-03-24 21:54:43,449 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245b29a5cd55ac468c8f3afa4b59619c3c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245b29a5cd55ac468c8f3afa4b59619c3c 2023-03-24 21:54:43,451 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032485e428addc274d5385ad7e7300a2d502 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032485e428addc274d5385ad7e7300a2d502 2023-03-24 21:54:43,452 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ba27615060341debbe917a955425b54 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248ba27615060341debbe917a955425b54 2023-03-24 21:54:43,454 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032491d88f7b10364474a2ae283ca28ea59c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032491d88f7b10364474a2ae283ca28ea59c 2023-03-24 21:54:43,455 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324965b7927a34444b58cbc1aa0d0491ea4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324965b7927a34444b58cbc1aa0d0491ea4 2023-03-24 21:54:43,456 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c908c8299504e0596adf468f4998484 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249c908c8299504e0596adf468f4998484 2023-03-24 21:54:43,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a00c0878771740fdb15405a9c4377f27 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a00c0878771740fdb15405a9c4377f27 2023-03-24 21:54:43,464 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a3faaa50e27d4527be22edcab44a3c53 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a3faaa50e27d4527be22edcab44a3c53 2023-03-24 21:54:43,465 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a4d11765463d446ea91fa71c308a959b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a4d11765463d446ea91fa71c308a959b 2023-03-24 21:54:43,466 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b2bf1a60552141e0b295b25fe70908cb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b2bf1a60552141e0b295b25fe70908cb 2023-03-24 21:54:43,468 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9488e20d9234fc2add966e9888fb407 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9488e20d9234fc2add966e9888fb407 2023-03-24 21:54:43,468 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:54:43,471 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,473 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:54:43,475 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:54:43,476 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,476 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:54:43,477 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694883476"}]},"ts":"9223372036854775807"} 2023-03-24 21:54:43,478 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:54:43,478 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 733bfd60f65777910098447a7a228a5c, NAME => 'TestAcidGuarantees,,1679694849672.733bfd60f65777910098447a7a228a5c.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:54:43,478 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:54:43,479 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694883478"}]},"ts":"9223372036854775807"} 2023-03-24 21:54:43,480 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:54:43,487 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 83 msec 2023-03-24 21:54:43,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-24 21:54:43,507 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-24 21:54:43,522 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 234), OpenFileDescriptor=525 (was 563), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=358 (was 370), ProcessCount=169 (was 168) - ProcessCount LEAK? -, AvailableMemoryMB=8128 (was 8392) 2023-03-24 21:54:43,539 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=525, MaxFileDescriptor=60000, SystemLoadAverage=358, ProcessCount=169, AvailableMemoryMB=8128 2023-03-24 21:54:43,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:54:43,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:54:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:54:43,545 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:54:43,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-24 21:54:43,546 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:54:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 21:54:43,547 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:43,548 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe empty. 2023-03-24 21:54:43,548 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:43,548 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:54:43,578 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:54:43,586 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => f67314e16f9680bd83bac15d2c4afabe, NAME => 'TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:54:43,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 21:54:43,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 21:54:44,001 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:54:44,001 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing f67314e16f9680bd83bac15d2c4afabe, disabling compactions & flushes 2023-03-24 21:54:44,001 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,001 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,001 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. after waiting 0 ms 2023-03-24 21:54:44,001 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,001 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,002 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:44,004 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:54:44,005 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694884005"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694884005"}]},"ts":"1679694884005"} 2023-03-24 21:54:44,006 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:54:44,007 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:54:44,007 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694884007"}]},"ts":"1679694884007"} 2023-03-24 21:54:44,008 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:54:44,011 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, ASSIGN}] 2023-03-24 21:54:44,012 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, ASSIGN 2023-03-24 21:54:44,013 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:54:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 21:54:44,164 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f67314e16f9680bd83bac15d2c4afabe, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,164 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694884164"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694884164"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694884164"}]},"ts":"1679694884164"} 2023-03-24 21:54:44,166 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:54:44,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => f67314e16f9680bd83bac15d2c4afabe, NAME => 'TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:54:44,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:54:44,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,323 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,324 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,326 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A 2023-03-24 21:54:44,326 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A 2023-03-24 21:54:44,326 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:44,327 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f67314e16f9680bd83bac15d2c4afabe columnFamilyName A 2023-03-24 21:54:44,327 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(310): Store=f67314e16f9680bd83bac15d2c4afabe/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:44,327 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,328 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B 2023-03-24 21:54:44,328 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B 2023-03-24 21:54:44,328 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:44,329 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f67314e16f9680bd83bac15d2c4afabe columnFamilyName B 2023-03-24 21:54:44,329 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(310): Store=f67314e16f9680bd83bac15d2c4afabe/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:44,329 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,330 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C 2023-03-24 21:54:44,330 DEBUG [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C 2023-03-24 21:54:44,330 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:54:44,330 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region f67314e16f9680bd83bac15d2c4afabe columnFamilyName C 2023-03-24 21:54:44,331 INFO [StoreOpener-f67314e16f9680bd83bac15d2c4afabe-1] regionserver.HStore(310): Store=f67314e16f9680bd83bac15d2c4afabe/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:54:44,331 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,331 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,332 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,333 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:54:44,334 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:54:44,336 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened f67314e16f9680bd83bac15d2c4afabe; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64092385, jitterRate=-0.044949039816856384}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:54:44,336 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:44,337 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., pid=30, masterSystemTime=1679694884317 2023-03-24 21:54:44,338 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,338 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,338 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=f67314e16f9680bd83bac15d2c4afabe, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,339 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694884338"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694884338"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694884338"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694884338"}]},"ts":"1679694884338"} 2023-03-24 21:54:44,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-24 21:54:44,340 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 in 173 msec 2023-03-24 21:54:44,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-24 21:54:44,342 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, ASSIGN in 329 msec 2023-03-24 21:54:44,342 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:54:44,342 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694884342"}]},"ts":"1679694884342"} 2023-03-24 21:54:44,343 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:54:44,345 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:54:44,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 803 msec 2023-03-24 21:54:44,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-24 21:54:44,649 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-24 21:54:44,650 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1594c869 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,656 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@463c0f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,657 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,658 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52572, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,659 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:54:44,660 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:42712, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:54:44,661 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x18111197 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,665 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2b75457c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,665 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x7a1b30ad to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,668 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a161bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,669 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x018daa76 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,674 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@22306cb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,674 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5c25b245 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,679 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30976dad, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,679 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x7875eca1 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,682 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@933de70, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,682 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5a7e7bef to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,686 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d29ee8e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,686 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1f775631 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,693 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@367ff733, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,693 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x156e01b7 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,697 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@300d1bf5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,698 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x4c004e36 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,701 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29068db8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,701 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1d47b4b3 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:54:44,705 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53f828d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:54:44,713 DEBUG [hconnection-0x503ccc5e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,714 DEBUG [hconnection-0x77225170-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,714 DEBUG [hconnection-0x4fb121bc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,715 DEBUG [hconnection-0x3eaf68fb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,715 DEBUG [hconnection-0x5f9f91f7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,715 DEBUG [hconnection-0x324f476c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,716 DEBUG [hconnection-0x774ab81-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,717 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52592, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,717 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52604, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,717 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52580, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,717 DEBUG [hconnection-0x6ea852cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:44,719 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52612, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,719 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52626, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,719 DEBUG [hconnection-0x54489ca2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,719 DEBUG [hconnection-0x7f6df26-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:54:44,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52642, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52622, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,723 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52652, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,723 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52660, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,727 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:52668, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:54:44,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:44,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:44,730 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:44,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:44,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:44,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:44,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:44,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:44,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:44,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,733 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:44,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:44,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:44,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:44,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:44,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:44,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:44,742 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:44,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:44,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:54:44,743 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:44,744 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:44,744 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:44,744 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:44,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:44,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:44,744 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:44,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:44,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,751 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,752 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,752 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:44,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3021f82f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:44,754 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:44,754 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,755 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:44,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:44,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:44,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:44,757 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:44,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:44,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:44,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:44,760 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:44,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:44,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:44,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:44,760 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:44,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:44,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694944767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694944768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694944768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694944769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:44,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:44,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:44,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:44,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,773 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:44,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:44,773 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,773 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:44,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694944771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,873 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:44,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694944872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:44,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694944872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694944873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694944872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:44,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:44,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694944875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:44,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:44,879 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:44,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:44,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:44,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:44,880 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,880 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,881 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:44,881 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:44,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,882 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,882 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:44,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:44,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:44,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:44,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:44,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:44,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:44,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:44,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:44,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:44,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:44,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,889 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,889 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,889 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:44,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ef28d01[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:44,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:44,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,892 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:44,892 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:44,892 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:44,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:44,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:44,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:44,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:44,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:44,897 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:44,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:44,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:44,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:44,897 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:44,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:44,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:44,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:44,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:44,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:44,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:44,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:44,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:44,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:44,904 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:44,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:44,904 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:44,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:44,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,004 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,007 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,008 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,008 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,010 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,010 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,011 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,011 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,012 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,012 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,012 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,012 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,012 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,015 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,015 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,015 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37e62763[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,018 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,024 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,024 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,030 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,030 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694945074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694945075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694945076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694945076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,081 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694945079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,130 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,134 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,134 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,136 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,137 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,137 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,137 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,137 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,138 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,138 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,138 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,138 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,139 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,140 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,140 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,140 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26431850[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,144 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,145 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,148 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,148 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,154 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,154 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,165 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/7eee5321a90d473987630e91fa8358ad 2023-03-24 21:54:45,254 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,257 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,260 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,260 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,260 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,261 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,261 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,261 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,261 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,262 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@199c6c37[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,266 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,266 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,268 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,270 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,270 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,277 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,277 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,377 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,379 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694945378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,380 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694945379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694945380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694945381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,382 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694945382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,384 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,384 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,384 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,384 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33ac6080[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,393 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,394 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,395 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,402 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,402 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,503 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,506 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,519 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,519 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,519 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,519 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,519 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,519 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,519 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,519 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,519 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,520 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,520 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@357784a9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,525 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,525 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,526 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,529 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,531 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,532 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,538 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,538 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,538 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,581 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/80cc0bcf10c44df086f6a1838de31fb7 2023-03-24 21:54:45,638 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,642 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,643 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,643 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,645 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,646 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,646 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,646 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,646 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,646 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,649 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b489bcf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,655 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,655 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,655 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,656 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,656 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,658 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,659 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,662 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,662 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,662 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,662 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,662 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,763 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,768 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,770 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,771 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,772 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,772 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,772 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,772 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,773 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71cfa345[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,786 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,786 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,786 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,789 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,789 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,790 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,802 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,802 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,803 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,803 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694945882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694945883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694945884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,888 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694945886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:45,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694945888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,902 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:45,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:45,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:45,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:45,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:45,905 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:45,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:45,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:45,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,909 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:45,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:45,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:45,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:45,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:45,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:45,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:45,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:45,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:45,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:45,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:45,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:45,917 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2137759c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:45,917 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:45,917 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:45,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:45,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:45,920 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:45,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:45,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:45,922 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:45,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:45,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:45,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:45,922 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:45,926 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,926 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,926 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:45,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,927 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:45,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:45,928 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:45,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:45,931 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:45,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:45,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:45,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:45,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:45,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:45,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:45,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:45,931 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:45,931 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:45,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/83e8a94ae11941e0ad13bc8458d24bf8 2023-03-24 21:54:46,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/7eee5321a90d473987630e91fa8358ad as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad 2023-03-24 21:54:46,005 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad, entries=200, sequenceid=14, filesize=13.9 K 2023-03-24 21:54:46,005 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/80cc0bcf10c44df086f6a1838de31fb7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7 2023-03-24 21:54:46,009 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 21:54:46,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/83e8a94ae11941e0ad13bc8458d24bf8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8 2023-03-24 21:54:46,015 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8, entries=150, sequenceid=14, filesize=11.6 K 2023-03-24 21:54:46,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for f67314e16f9680bd83bac15d2c4afabe in 1272ms, sequenceid=14, compaction requested=false 2023-03-24 21:54:46,016 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,032 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,035 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,037 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,038 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,038 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,039 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:46,040 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,050 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/f2863931f49f43bcaa1d9fa7e345b714 2023-03-24 21:54:46,060 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/7254cb6c82cf43db8168b613d251e9e0 2023-03-24 21:54:46,087 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/73f12b58f1b448ab8663d5ec24894bb1 2023-03-24 21:54:46,100 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/f2863931f49f43bcaa1d9fa7e345b714 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714 2023-03-24 21:54:46,107 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714, entries=150, sequenceid=33, filesize=11.6 K 2023-03-24 21:54:46,109 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/7254cb6c82cf43db8168b613d251e9e0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0 2023-03-24 21:54:46,113 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0, entries=150, sequenceid=33, filesize=11.6 K 2023-03-24 21:54:46,114 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/73f12b58f1b448ab8663d5ec24894bb1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1 2023-03-24 21:54:46,121 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1, entries=150, sequenceid=33, filesize=11.6 K 2023-03-24 21:54:46,121 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 82ms, sequenceid=33, compaction requested=false 2023-03-24 21:54:46,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28ecffdf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,132 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,136 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,137 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,137 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,137 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,137 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,139 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,139 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,139 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,148 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,151 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,152 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,152 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,252 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,258 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,261 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,263 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,265 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,265 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,265 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,265 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,265 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b58fee[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,272 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,278 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,279 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,284 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,288 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,288 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,388 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,392 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,396 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6beccdd6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,409 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,413 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,413 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,413 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,413 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,413 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,420 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,420 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,420 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,420 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,420 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,421 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,421 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,421 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,421 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,421 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,421 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,421 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,521 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,526 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@716dcea2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,535 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,537 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,537 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,537 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,537 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,540 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,540 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,546 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,546 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,546 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,646 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,650 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,653 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,654 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,654 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,654 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,654 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,659 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,659 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e30d557[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,660 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,660 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,660 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,660 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,660 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,660 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,661 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,661 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,661 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,661 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,664 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,664 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,666 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,666 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,666 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,666 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,671 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,671 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,771 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,775 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,777 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:46,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,780 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75b1e780[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,785 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,786 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,786 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,788 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,789 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,790 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,790 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,790 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,795 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,795 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:46,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:54:46,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:46,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:46,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,895 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:46,895 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:46,895 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:46,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:46,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:46,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:46,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:46,898 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:46,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:46,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:46,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:46,902 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:46,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:46,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:46,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:46,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:46,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:46,904 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,904 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:46,904 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:46,904 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:46,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:46,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,905 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:46,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@783e169d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:46,908 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:46,908 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,909 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:46,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:46,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:46,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:46,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:46,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:46,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:46,912 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:46,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:46,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:46,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:46,912 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:46,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:46,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,917 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:46,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:46,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:46,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:46,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:46,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:46,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:46,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:46,919 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:46,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:46,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:46,920 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:46,920 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:46,939 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:46,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694946935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694946938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:46,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694946938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:46,943 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694946939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:46,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694946939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,020 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,023 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,025 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,027 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6739dfe6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,032 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,033 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,033 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,035 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,038 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,038 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,040 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,040 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694947040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,044 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,044 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:47,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694947042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694947043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694947044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694947045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,144 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,147 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,150 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1b243b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,156 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,166 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,166 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,172 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:47,172 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,172 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694947243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,247 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694947246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694947247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694947247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694947248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,273 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,277 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,279 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,280 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,281 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,281 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,281 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,281 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@256c55bf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,290 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,300 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,300 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,314 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,313 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,314 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,314 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,318 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/1c498924f7414bceb723fe6e51ab2696 2023-03-24 21:54:47,414 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,417 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,419 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,419 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,419 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,419 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,420 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,420 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,420 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,421 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,421 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,421 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,421 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,421 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,423 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,423 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d6e2571[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,425 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,430 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,430 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,432 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,432 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,432 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,436 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,436 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,536 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,539 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,542 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,542 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,543 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,543 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,543 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,543 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,543 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,546 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,546 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,546 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40119d5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,547 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,547 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694947546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,549 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694947550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694947550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,551 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694947550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,554 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,554 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,554 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,556 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,556 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,556 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,557 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:47,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,559 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694947557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,569 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,569 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,669 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,672 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,672 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,672 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,672 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,673 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:47,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,676 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,676 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,676 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,676 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,676 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,677 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,677 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,678 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,678 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ddf92[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,680 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,681 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,681 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,683 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,685 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,685 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,685 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,685 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,685 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,691 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,691 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,730 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/a59daa275fab4e95800221589877622c 2023-03-24 21:54:47,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/2bc173268b774a199d7d5ddb221399d3 2023-03-24 21:54:47,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/1c498924f7414bceb723fe6e51ab2696 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696 2023-03-24 21:54:47,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 21:54:47,752 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/a59daa275fab4e95800221589877622c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c 2023-03-24 21:54:47,757 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 21:54:47,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/2bc173268b774a199d7d5ddb221399d3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3 2023-03-24 21:54:47,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3, entries=150, sequenceid=50, filesize=11.6 K 2023-03-24 21:54:47,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f67314e16f9680bd83bac15d2c4afabe in 869ms, sequenceid=50, compaction requested=true 2023-03-24 21:54:47,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:47,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:47,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:47,763 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:47,763 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:47,763 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:47,764 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:47,764 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:47,764 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,764 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.8 K 2023-03-24 21:54:47,765 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 80cc0bcf10c44df086f6a1838de31fb7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694884732 2023-03-24 21:54:47,765 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:47,765 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:47,765 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,766 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=37.1 K 2023-03-24 21:54:47,766 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7254cb6c82cf43db8168b613d251e9e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679694884753 2023-03-24 21:54:47,766 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7eee5321a90d473987630e91fa8358ad, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694884732 2023-03-24 21:54:47,767 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a59daa275fab4e95800221589877622c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:47,767 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f2863931f49f43bcaa1d9fa7e345b714, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679694884753 2023-03-24 21:54:47,767 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1c498924f7414bceb723fe6e51ab2696, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:47,775 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#207 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:47,780 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#208 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:47,791 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:47,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:47,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:47,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:47,811 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:47,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:47,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:47,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,813 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:47,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:47,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:47,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:47,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:47,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:47,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:47,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:47,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:47,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:47,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:47,823 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 21:54:47,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:47,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:47,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:47,824 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:47,824 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:47,824 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:47,828 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/ead03957bc2742bfb341813291b80bc6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/ead03957bc2742bfb341813291b80bc6 2023-03-24 21:54:47,835 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into ead03957bc2742bfb341813291b80bc6(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:47,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:47,835 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=13, startTime=1679694887763; duration=0sec 2023-03-24 21:54:47,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:47,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:47,837 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:47,837 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:47,837 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,837 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.8 K 2023-03-24 21:54:47,838 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 83e8a94ae11941e0ad13bc8458d24bf8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679694884732 2023-03-24 21:54:47,838 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 73f12b58f1b448ab8663d5ec24894bb1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679694884753 2023-03-24 21:54:47,838 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2bc173268b774a199d7d5ddb221399d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:47,844 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#210 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:47,846 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/df53c9134c204c3d8814e4cce4e58a57 2023-03-24 21:54:47,883 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/47bac44890f94352b8334786d5a8a583 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/47bac44890f94352b8334786d5a8a583 2023-03-24 21:54:47,885 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f3959dc8059e4ae08eec881df951349b 2023-03-24 21:54:47,889 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 47bac44890f94352b8334786d5a8a583(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:47,889 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:47,889 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=13, startTime=1679694887763; duration=0sec 2023-03-24 21:54:47,889 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:47,905 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/ef20ce46513342eeb6937f6bcd514574 2023-03-24 21:54:47,910 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/df53c9134c204c3d8814e4cce4e58a57 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57 2023-03-24 21:54:47,914 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 21:54:47,915 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f3959dc8059e4ae08eec881df951349b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b 2023-03-24 21:54:47,922 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 21:54:47,923 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/ef20ce46513342eeb6937f6bcd514574 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574 2023-03-24 21:54:47,931 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574, entries=150, sequenceid=70, filesize=11.6 K 2023-03-24 21:54:47,932 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 109ms, sequenceid=70, compaction requested=false 2023-03-24 21:54:47,932 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:47,932 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:47,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:47,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:47,933 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,935 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:47,935 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:47,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,938 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,939 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,939 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:47,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:47,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45bbb3c7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:47,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:47,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:47,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:47,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:47,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:47,945 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:47,945 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,945 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:47,945 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:47,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:47,950 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:47,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:47,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:47,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:47,950 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:47,952 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,952 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:47,952 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:47,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:47,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:47,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:47,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:47,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:47,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:47,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:47,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:47,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:47,959 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:47,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:47,959 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:47,959 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:47,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:47,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:48,058 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:48,058 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:48,060 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,070 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,075 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694888073, max timestamp=1679694888080Num uniques 859; 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694888073, max timestamp=1679694888080Num uniques 859; 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694888073, max timestamp=1679694888080Num uniques 859; 2023-03-24 21:54:48,082 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:54:48,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,089 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/06421227de0e45b2bf265e451c4f7488 2023-03-24 21:54:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@192253ac[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,093 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,101 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,101 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,104 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694948100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694948102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694948103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694948103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694948104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,118 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:48,118 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,119 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/dacdbe81a93e4a8282c69a0d227add59 2023-03-24 21:54:48,149 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=82 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8dbcefe8f80840919ac054202812f27d 2023-03-24 21:54:48,155 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/06421227de0e45b2bf265e451c4f7488 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488 2023-03-24 21:54:48,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 21:54:48,164 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/dacdbe81a93e4a8282c69a0d227add59 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59 2023-03-24 21:54:48,168 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 21:54:48,169 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8dbcefe8f80840919ac054202812f27d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d 2023-03-24 21:54:48,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d, entries=150, sequenceid=82, filesize=11.6 K 2023-03-24 21:54:48,175 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f67314e16f9680bd83bac15d2c4afabe in 117ms, sequenceid=82, compaction requested=true 2023-03-24 21:54:48,175 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:48,175 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:48,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:48,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:48,176 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:54:48,176 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:48,176 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:48,176 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,176 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/ead03957bc2742bfb341813291b80bc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.9 K 2023-03-24 21:54:48,177 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ead03957bc2742bfb341813291b80bc6, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:48,179 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting df53c9134c204c3d8814e4cce4e58a57, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679694886937 2023-03-24 21:54:48,179 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 06421227de0e45b2bf265e451c4f7488, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:48,184 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#216 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:48,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:48,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-24 21:54:48,210 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:48,210 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:48,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:48,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:48,211 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:48,211 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:48,218 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/5fe5523538824490924248a3122b0bc3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/5fe5523538824490924248a3122b0bc3 2023-03-24 21:54:48,218 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694948220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,226 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into 5fe5523538824490924248a3122b0bc3(size=11.7 K), total size for store is 34.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:48,226 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:48,226 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=13, startTime=1679694887763; duration=0sec 2023-03-24 21:54:48,226 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:54:48,226 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:48,227 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:48,227 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:48,227 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,227 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/47bac44890f94352b8334786d5a8a583, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.9 K 2023-03-24 21:54:48,228 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 47bac44890f94352b8334786d5a8a583, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:48,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694948223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,229 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ef20ce46513342eeb6937f6bcd514574, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679694886937 2023-03-24 21:54:48,229 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8dbcefe8f80840919ac054202812f27d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:48,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694948225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694948225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,232 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694948226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,234 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,235 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,236 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#218 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e13eec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,255 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,255 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,264 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/17de76a7534a4414bccfdc71146e2fd9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/17de76a7534a4414bccfdc71146e2fd9 2023-03-24 21:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,266 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,266 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,266 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,269 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 17de76a7534a4414bccfdc71146e2fd9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:48,269 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:48,269 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=13, startTime=1679694888176; duration=0sec 2023-03-24 21:54:48,270 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:48,270 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:48,271 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:48,271 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:48,271 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,271 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/5fe5523538824490924248a3122b0bc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.9 K 2023-03-24 21:54:48,271 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5fe5523538824490924248a3122b0bc3, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679694886888 2023-03-24 21:54:48,272 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f3959dc8059e4ae08eec881df951349b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679694886937 2023-03-24 21:54:48,272 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting dacdbe81a93e4a8282c69a0d227add59, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:48,278 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#219 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:48,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694948327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694948329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694948333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694948333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694948333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,366 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,370 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,372 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,373 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,374 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,374 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,374 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,374 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,374 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,374 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,376 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79cca28d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,380 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,382 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,382 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,385 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,390 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,390 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,490 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,494 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,495 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,495 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,497 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,500 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@617103f8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,512 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,516 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,516 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,516 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,522 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,522 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,522 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694948530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694948530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694948536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694948536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694948537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,607 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/27572c5504da4bbbbc027a933b0a136c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/27572c5504da4bbbbc027a933b0a136c 2023-03-24 21:54:48,612 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into 27572c5504da4bbbbc027a933b0a136c(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:48,612 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:48,612 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=13, startTime=1679694888175; duration=0sec 2023-03-24 21:54:48,612 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:48,622 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,625 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,631 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,633 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,633 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,633 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,635 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a459349[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,638 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,638 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,642 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,647 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,647 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,653 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:48,653 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/9a6709c9c8334db2a852affefe4f3ba1 2023-03-24 21:54:48,686 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/47e21716f5c34944aa3e02f03b2d15ec as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/47e21716f5c34944aa3e02f03b2d15ec 2023-03-24 21:54:48,698 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into 47e21716f5c34944aa3e02f03b2d15ec(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:48,698 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:48,698 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=13, startTime=1679694888176; duration=0sec 2023-03-24 21:54:48,698 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:48,753 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,754 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:54:48,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,757 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,764 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,774 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,774 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76cfd6b3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,777 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,782 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,782 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,785 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,792 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:48,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,792 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694948833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694948834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694948840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694948842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:48,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694948842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,892 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:48,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:48,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:48,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:48,896 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:48,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:48,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:48,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:48,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:48,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:48,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:48,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:48,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a0bda2d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:48,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:48,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:48,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:48,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:48,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:48,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:48,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:48,911 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:48,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:48,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:48,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:48,911 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:48,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,912 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:48,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:48,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:48,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:48,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:48,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:48,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:48,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:48,919 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:48,919 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:48,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:48,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,019 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,022 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,023 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,025 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,025 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,025 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:49,025 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c6a57[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,033 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,033 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,033 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,037 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,037 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,049 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,049 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f6d4a6e5413d4d72b17488c1a6640d9b 2023-03-24 21:54:49,149 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,153 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:49,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,158 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45957ea4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,166 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,167 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,176 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,176 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,176 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,176 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,176 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,177 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,177 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:49,177 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,277 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,284 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,286 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,286 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,286 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,286 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:49,286 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d6752eb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,290 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,294 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,294 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,301 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,301 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,301 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,301 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:49,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694949337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:49,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694949341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:49,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694949343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,348 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:49,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694949346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:49,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694949346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,401 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,405 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,408 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,408 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,409 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,409 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,409 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,409 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,409 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:49,409 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790933d8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,416 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,416 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,416 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,425 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,425 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,435 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,435 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,489 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=118 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/e3de1a3edf824bd9bb8db3b95e4c9868 2023-03-24 21:54:49,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/9a6709c9c8334db2a852affefe4f3ba1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1 2023-03-24 21:54:49,503 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1, entries=200, sequenceid=118, filesize=13.9 K 2023-03-24 21:54:49,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f6d4a6e5413d4d72b17488c1a6640d9b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b 2023-03-24 21:54:49,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b, entries=150, sequenceid=118, filesize=11.6 K 2023-03-24 21:54:49,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/e3de1a3edf824bd9bb8db3b95e4c9868 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868 2023-03-24 21:54:49,512 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868, entries=150, sequenceid=118, filesize=11.6 K 2023-03-24 21:54:49,513 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=53.67 KB/54960 for f67314e16f9680bd83bac15d2c4afabe in 1303ms, sequenceid=118, compaction requested=false 2023-03-24 21:54:49,513 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:49,535 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,538 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,540 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,541 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,541 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,541 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,541 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,542 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,542 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,542 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:49,542 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:49,550 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/15fb8571236f4f429b316d31b5f1d272 2023-03-24 21:54:49,560 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/26c4d9ad702b4f5dbfb43c883c292d56 2023-03-24 21:54:49,576 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=133 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/d744729fe57a48b991fbf3bee6450ab6 2023-03-24 21:54:49,580 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/15fb8571236f4f429b316d31b5f1d272 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272 2023-03-24 21:54:49,584 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272, entries=150, sequenceid=133, filesize=11.6 K 2023-03-24 21:54:49,585 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/26c4d9ad702b4f5dbfb43c883c292d56 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56 2023-03-24 21:54:49,592 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56, entries=150, sequenceid=133, filesize=11.6 K 2023-03-24 21:54:49,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/d744729fe57a48b991fbf3bee6450ab6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6 2023-03-24 21:54:49,602 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6, entries=150, sequenceid=133, filesize=11.6 K 2023-03-24 21:54:49,603 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 61ms, sequenceid=133, compaction requested=true 2023-03-24 21:54:49,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:49,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,605 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,606 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,606 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,606 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b7a07c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,609 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,609 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,609 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,609 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,610 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,613 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,614 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,621 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,621 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,721 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,725 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,726 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,726 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,727 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,727 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,727 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,728 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,728 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:49,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,731 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b99db0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,733 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,733 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,734 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,734 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,736 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,739 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,739 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,742 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,746 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,746 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,746 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,746 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:49,846 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,850 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,853 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,853 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,855 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,856 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:49,856 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e69a6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,862 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,864 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:49,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:49,868 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:49,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:49,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:49,868 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:49,868 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:49,873 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,873 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,873 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,874 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,882 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:49,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:49,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,882 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:49,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:49,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:49,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:49,882 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:49,983 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:49,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:49,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:49,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:49,986 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:49,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:49,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:49,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,988 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:49,988 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:49,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:49,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:49,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:49,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:49,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:49,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:49,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:49,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:49,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:49,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:49,990 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:49,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:49,991 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:49,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:49,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:49,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:49,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:49,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:49,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:49,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30190d8a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:49,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:49,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:49,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:49,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:49,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:49,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:49,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:49,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:49,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:49,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,003 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,003 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,009 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,009 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,109 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,113 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:50,117 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,119 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1409efb5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,123 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,124 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,127 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,127 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,134 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,134 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,234 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,237 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,241 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,241 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,241 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,241 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:50,241 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fe6a499[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,252 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,252 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,262 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,262 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,262 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,262 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:50,354 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:50,356 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:50,362 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,375 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,377 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,377 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,386 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,386 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,387 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:50,387 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c669b9a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,400 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,400 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,407 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,407 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,423 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694950418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694890404, max timestamp=1679694890422Num uniques 859; 2023-03-24 21:54:50,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694890404, max timestamp=1679694890422Num uniques 859; 2023-03-24 21:54:50,424 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679694890404, max timestamp=1679694890422Num uniques 859; 2023-03-24 21:54:50,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:50,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:50,425 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-24 21:54:50,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694950435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694950437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694950438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694950438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,507 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,510 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,512 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,513 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,513 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,514 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,514 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,514 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,515 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,515 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,516 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:50,516 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,516 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,516 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,516 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,516 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,516 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@246bb8c1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,524 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,524 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694950524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,531 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,531 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,531 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694950539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694950541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694950543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694950544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,631 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,635 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,637 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:50,639 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f448fbe[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,650 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,653 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,653 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,658 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,658 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,658 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,658 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694950727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694950743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694950745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694950748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:50,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694950749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,759 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,762 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,767 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,768 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,769 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,769 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,769 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:50,769 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@651c2ea3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,773 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,774 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,775 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,776 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,779 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,779 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,788 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/22d72b7bd240498c86dfccf2d184cdf7 2023-03-24 21:54:50,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,799 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,799 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,799 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f295d13b7ea44669885f56869cacd79c 2023-03-24 21:54:50,899 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:50,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:50,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:50,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:50,903 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:50,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:50,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:50,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,905 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:50,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,906 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:50,907 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:50,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:50,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:50,907 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:50,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:50,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:50,916 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,916 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:50,916 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:50,916 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:50,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:50,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:50,916 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,917 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:50,917 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:50,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,919 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,919 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:50,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dcec3d1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:50,921 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:50,921 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,923 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:50,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:50,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:50,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:50,924 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,925 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:50,925 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,925 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:50,928 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:50,928 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:50,928 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:50,928 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:50,928 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:50,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:50,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:50,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:50,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:50,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:50,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:50,935 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:50,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:50,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:50,936 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,936 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694951030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,036 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:51,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:51,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:51,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:51,039 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:51,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:51,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:51,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,040 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:51,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:51,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:51,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:51,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:51,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:51,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:51,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:51,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:51,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:51,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:51,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:51,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:51,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:51,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:51,046 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,046 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,046 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:51,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@741acf19[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:51,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:51,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:51,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:51,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:51,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:51,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:51,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694951049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694951049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694951051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:51,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:51,054 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:51,054 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:51,054 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:51,054 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:51,054 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:51,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694951053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:51,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:51,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:51,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:51,060 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:51,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:51,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:51,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:51,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,060 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:51,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,161 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:51,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:51,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:51,164 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:51,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:51,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:51,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:51,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:51,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:51,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:51,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:51,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:51,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:51,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:51,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:51,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:51,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,173 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,173 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:51,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f78975b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:51,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:51,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:51,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:51,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:51,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:51,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:51,180 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:51,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:51,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:51,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:51,180 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:51,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:51,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:51,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,183 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:51,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:51,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:51,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:51,187 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:51,187 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:51,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:51,187 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:51,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:51,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:51,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/720e253c0f7e489a9909243161f7bcf9 2023-03-24 21:54:51,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/22d72b7bd240498c86dfccf2d184cdf7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7 2023-03-24 21:54:51,224 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7, entries=200, sequenceid=150, filesize=14.1 K 2023-03-24 21:54:51,225 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f295d13b7ea44669885f56869cacd79c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c 2023-03-24 21:54:51,228 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c, entries=150, sequenceid=150, filesize=11.8 K 2023-03-24 21:54:51,228 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/720e253c0f7e489a9909243161f7bcf9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9 2023-03-24 21:54:51,232 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9, entries=150, sequenceid=150, filesize=11.8 K 2023-03-24 21:54:51,232 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for f67314e16f9680bd83bac15d2c4afabe in 878ms, sequenceid=150, compaction requested=true 2023-03-24 21:54:51,233 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:51,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:51,233 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:51,233 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:51,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:51,233 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:51,234 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:51,234 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:51,234 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,234 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/27572c5504da4bbbbc027a933b0a136c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=51.4 K 2023-03-24 21:54:51,234 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:51,234 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:51,234 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,235 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/47e21716f5c34944aa3e02f03b2d15ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=46.8 K 2023-03-24 21:54:51,235 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 27572c5504da4bbbbc027a933b0a136c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:51,235 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 47e21716f5c34944aa3e02f03b2d15ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:51,235 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9a6709c9c8334db2a852affefe4f3ba1, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694888101 2023-03-24 21:54:51,236 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 15fb8571236f4f429b316d31b5f1d272, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679694888223 2023-03-24 21:54:51,236 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f6d4a6e5413d4d72b17488c1a6640d9b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694888101 2023-03-24 21:54:51,236 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 22d72b7bd240498c86dfccf2d184cdf7, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890352 2023-03-24 21:54:51,236 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 26c4d9ad702b4f5dbfb43c883c292d56, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679694888223 2023-03-24 21:54:51,236 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f295d13b7ea44669885f56869cacd79c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890353 2023-03-24 21:54:51,245 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#228 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:51,248 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#229 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:51,283 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/cc9794eeaeb14292b9cce778c266571b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/cc9794eeaeb14292b9cce778c266571b 2023-03-24 21:54:51,287 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:51,287 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into cc9794eeaeb14292b9cce778c266571b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:51,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:51,287 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:51,287 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=12, startTime=1679694891233; duration=0sec 2023-03-24 21:54:51,288 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:51,288 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:51,289 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:51,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:51,290 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:51,290 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,290 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/17de76a7534a4414bccfdc71146e2fd9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=46.8 K 2023-03-24 21:54:51,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:51,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:51,292 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:51,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:51,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:51,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,293 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 17de76a7534a4414bccfdc71146e2fd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=82, earliestPutTs=1679694888056 2023-03-24 21:54:51,293 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e3de1a3edf824bd9bb8db3b95e4c9868, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=118, earliestPutTs=1679694888101 2023-03-24 21:54:51,294 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:51,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:51,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,294 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d744729fe57a48b991fbf3bee6450ab6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=133, earliestPutTs=1679694888223 2023-03-24 21:54:51,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:51,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,294 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 720e253c0f7e489a9909243161f7bcf9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890353 2023-03-24 21:54:51,295 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:51,295 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:51,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:51,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:51,296 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:51,296 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:51,296 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:51,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:51,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:51,303 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:51,303 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:51,304 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:51,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:51,304 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:51,322 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/53a12f1661b44c348ca4da84bcd4f0f2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/53a12f1661b44c348ca4da84bcd4f0f2 2023-03-24 21:54:51,327 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 53a12f1661b44c348ca4da84bcd4f0f2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:51,328 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:51,328 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=12, startTime=1679694891233; duration=0sec 2023-03-24 21:54:51,328 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:51,333 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/0a881d0a77c742989755ec2926d62402 2023-03-24 21:54:51,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:51,537 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:51,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694951556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694951556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694951557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694951557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694951560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,661 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694951661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694951661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694951661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694951664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,682 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/c340deb0daa646529004b53a5b979a96 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/c340deb0daa646529004b53a5b979a96 2023-03-24 21:54:51,686 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into c340deb0daa646529004b53a5b979a96(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:51,687 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:51,687 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=12, startTime=1679694891233; duration=0sec 2023-03-24 21:54:51,687 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:51,747 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/93121dcb8e8d4be69a470227ac08de4d 2023-03-24 21:54:51,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694951863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694951864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694951865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:51,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:51,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694951868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,158 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/7879c4957d844f859cb204b6309f2e87 2023-03-24 21:54:52,162 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/0a881d0a77c742989755ec2926d62402 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402 2023-03-24 21:54:52,166 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402, entries=150, sequenceid=184, filesize=11.8 K 2023-03-24 21:54:52,166 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/93121dcb8e8d4be69a470227ac08de4d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d 2023-03-24 21:54:52,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694952167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,170 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d, entries=150, sequenceid=184, filesize=11.8 K 2023-03-24 21:54:52,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694952167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,171 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/7879c4957d844f859cb204b6309f2e87 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87 2023-03-24 21:54:52,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694952170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694952171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,176 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87, entries=150, sequenceid=184, filesize=11.8 K 2023-03-24 21:54:52,177 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=60.38 KB/61830 for f67314e16f9680bd83bac15d2c4afabe in 873ms, sequenceid=184, compaction requested=false 2023-03-24 21:54:52,177 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:52,177 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:52,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:52,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:52,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:52,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:52,183 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:52,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:52,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b15dd21[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:52,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:52,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,188 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:52,188 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:52,188 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:52,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:52,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:52,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:52,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:52,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:52,193 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:52,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:52,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:52,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:52,193 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:52,195 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,195 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,195 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:52,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:52,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:52,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:52,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:52,200 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:52,201 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:52,301 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:52,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:52,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:52,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:52,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:52,304 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:52,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:52,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:52,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:52,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:52,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:52,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:52,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:52,308 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:52,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:52,315 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/149ecce992b3400db5ad37dfbf0f756d 2023-03-24 21:54:52,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:52,565 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:52,612 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694952611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694952671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694952672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694952673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,678 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694952676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694952713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,726 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/1ee89e500b8448b2b46a98b7d18e4700 2023-03-24 21:54:52,765 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/ae5861444de24de2a641cfb240bd1d74 2023-03-24 21:54:52,771 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/149ecce992b3400db5ad37dfbf0f756d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d 2023-03-24 21:54:52,777 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:54:52,777 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/1ee89e500b8448b2b46a98b7d18e4700 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700 2023-03-24 21:54:52,783 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:54:52,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/ae5861444de24de2a641cfb240bd1d74 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74 2023-03-24 21:54:52,793 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:54:52,794 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for f67314e16f9680bd83bac15d2c4afabe in 486ms, sequenceid=198, compaction requested=true 2023-03-24 21:54:52,794 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:52,794 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:52,794 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:52,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:52,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:52,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:52,796 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,796 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,797 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,797 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:52,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:52,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@503e53d9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:52,799 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:52,799 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,801 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,801 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:52,801 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:52,801 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:52,802 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,802 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:52,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:52,803 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:52,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:52,811 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:52,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:52,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:52,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:52,811 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:52,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:52,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:52,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:52,820 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:52,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:52,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:52,824 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:52,824 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:52,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:52,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:52,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:52,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694892603, max timestamp=1679694892918Num uniques 859; 2023-03-24 21:54:52,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:52,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:52,922 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=89.45 KB heapSize=262.11 KB 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:52,922 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:52,924 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:52,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:52,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:52,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:52,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:52,947 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:52,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:52,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:52,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:52,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,950 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:52,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:52,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:52,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:52,975 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:52,975 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:52,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:52,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:52,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:52,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:52,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694952973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:52,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:52,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:52,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:52,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:52,996 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:52,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:52,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:52,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:52,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@727e3cc0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,011 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,018 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,018 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,027 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,027 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:53,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694953077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,127 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,131 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,133 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,135 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,135 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,135 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,135 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,135 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,138 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19d7bf9b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,141 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,142 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,142 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,143 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,145 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,145 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:53,152 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,153 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,253 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,256 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,258 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,259 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,260 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,260 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,260 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,260 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,261 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,261 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2e9555[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,264 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,264 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,267 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,271 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:53,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,275 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,275 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694953280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,375 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,378 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,380 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/b9676ca9fc0f455a8282aaa2eb6397ac 2023-03-24 21:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e2cad92[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,393 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,393 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,398 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,401 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,404 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,404 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:53,504 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,519 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d371eab[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,529 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,532 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,532 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,536 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,540 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,540 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52652 deadline: 1679694953584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,640 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,644 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,644 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,649 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,649 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,649 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,649 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,649 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,650 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,651 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,652 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,652 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19b855f1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,655 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,660 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,660 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,660 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,660 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,660 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,665 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,668 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,668 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,668 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694953677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694953678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694953682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:53,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694953692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,768 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,771 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,773 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,774 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,774 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,775 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,775 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,775 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,776 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,776 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:53,776 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:53,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:53,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:53,776 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:53,777 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,777 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,778 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:53,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7154ffae[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:53,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:53,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,782 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,782 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:53,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:53,783 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:53,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:53,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:53,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:53,791 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:53,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:53,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:53,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:53,791 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:53,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:53,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:53,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,798 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:53,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/e2953268359a42c99bf6448e58fd7b41 2023-03-24 21:54:53,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:53,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:53,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:53,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:53,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,806 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:53,806 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:53,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,806 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:53,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:53,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:53,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/352e131447b442a4a3c61f4a20fe812f 2023-03-24 21:54:53,819 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/b9676ca9fc0f455a8282aaa2eb6397ac as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac 2023-03-24 21:54:53,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac, entries=200, sequenceid=219, filesize=14.1 K 2023-03-24 21:54:53,825 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/e2953268359a42c99bf6448e58fd7b41 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41 2023-03-24 21:54:53,828 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41, entries=150, sequenceid=219, filesize=11.8 K 2023-03-24 21:54:53,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/352e131447b442a4a3c61f4a20fe812f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f 2023-03-24 21:54:53,833 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f, entries=150, sequenceid=219, filesize=11.8 K 2023-03-24 21:54:53,833 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~89.45 KB/91600, heapSize ~262.06 KB/268352, currentSize=87.22 KB/89310 for f67314e16f9680bd83bac15d2c4afabe in 912ms, sequenceid=219, compaction requested=true 2023-03-24 21:54:53,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:53,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:53,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:53,834 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:53,834 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:53,834 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50876 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:53,835 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:53,835 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/c340deb0daa646529004b53a5b979a96, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=47.3 K 2023-03-24 21:54:53,835 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,835 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/cc9794eeaeb14292b9cce778c266571b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=49.7 K 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c340deb0daa646529004b53a5b979a96, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890353 2023-03-24 21:54:53,835 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cc9794eeaeb14292b9cce778c266571b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890353 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 93121dcb8e8d4be69a470227ac08de4d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679694890432 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 0a881d0a77c742989755ec2926d62402, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679694890432 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1ee89e500b8448b2b46a98b7d18e4700, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694891543 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 149ecce992b3400db5ad37dfbf0f756d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694891543 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e2953268359a42c99bf6448e58fd7b41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892607 2023-03-24 21:54:53,836 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b9676ca9fc0f455a8282aaa2eb6397ac, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892603 2023-03-24 21:54:53,845 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#241 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:53,845 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:53,906 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:53,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:53,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:53,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:53,909 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:53,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:53,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:53,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,911 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:53,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:53,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:53,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:53,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:53,912 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:53,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:53,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:53,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:53,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:53,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:53,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:53,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:53,913 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:53,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:53,934 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/d0ea9a1d51a84ecdaaab099b87801869 2023-03-24 21:54:54,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:54,092 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:54,129 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:54,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694954127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694954230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,258 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/c75d488ba7c94a7bbcdaa7a51d21a071 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c75d488ba7c94a7bbcdaa7a51d21a071 2023-03-24 21:54:54,261 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/d0a00886123740658fc341f420d2f57d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d0a00886123740658fc341f420d2f57d 2023-03-24 21:54:54,262 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into c75d488ba7c94a7bbcdaa7a51d21a071(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:54,263 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:54,263 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=12, startTime=1679694893834; duration=0sec 2023-03-24 21:54:54,263 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:54,263 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:54,264 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48486 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:54,264 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:54,264 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,265 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/53a12f1661b44c348ca4da84bcd4f0f2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=47.3 K 2023-03-24 21:54:54,265 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 53a12f1661b44c348ca4da84bcd4f0f2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1679694890353 2023-03-24 21:54:54,265 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7879c4957d844f859cb204b6309f2e87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1679694890432 2023-03-24 21:54:54,265 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into d0a00886123740658fc341f420d2f57d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:54,265 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:54,265 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=12, startTime=1679694893834; duration=0sec 2023-03-24 21:54:54,266 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:54,266 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ae5861444de24de2a641cfb240bd1d74, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694891543 2023-03-24 21:54:54,266 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 352e131447b442a4a3c61f4a20fe812f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892607 2023-03-24 21:54:54,272 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:54,346 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/2c082ec855fc4cc4a3b0d03759853cb9 2023-03-24 21:54:54,356 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/b8d88e125b3745fcb4e5ba63efc55769 2023-03-24 21:54:54,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/d0ea9a1d51a84ecdaaab099b87801869 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869 2023-03-24 21:54:54,365 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 21:54:54,366 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/2c082ec855fc4cc4a3b0d03759853cb9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9 2023-03-24 21:54:54,370 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 21:54:54,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/b8d88e125b3745fcb4e5ba63efc55769 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769 2023-03-24 21:54:54,374 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769, entries=150, sequenceid=235, filesize=11.8 K 2023-03-24 21:54:54,375 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for f67314e16f9680bd83bac15d2c4afabe in 462ms, sequenceid=235, compaction requested=false 2023-03-24 21:54:54,375 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:54,375 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:54,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:54,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:54,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:54,377 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,377 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:54,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2acf9c2b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:54,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:54,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,385 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:54,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:54,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:54,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:54,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:54,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:54,394 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:54,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:54,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:54,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:54,394 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:54,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,399 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:54,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:54,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,407 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:54,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:54,407 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:54,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:54,437 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:54,437 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,465 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/0b8a4f7a4d8249f0b5faa1fd94438a2c 2023-03-24 21:54:54,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/4673a95902824d8da87d03ff2adc7f08 2023-03-24 21:54:54,490 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:54,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694954488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,508 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:54,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:54,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:54,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:54,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:54,524 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:54,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:54,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:54,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:54,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:54,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:54,528 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,528 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:54,528 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:54,528 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:54,528 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:54,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:54,531 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:54,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:54,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f881d9f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:54,533 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:54,533 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,534 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:54,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:54,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:54,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:54,536 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,536 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:54,536 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,536 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:54,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:54,539 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:54,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:54,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:54,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:54,539 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:54,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:54,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:54,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,550 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:54,550 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:54,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,594 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:54,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694954592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,650 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:54,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:54,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:54,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:54,655 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:54,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:54,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:54,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,657 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,657 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,657 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,657 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:54,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,658 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:54,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:54,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:54,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:54,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:54,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,663 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:54,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:54,671 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:54,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:54,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:54,672 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:54,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:54,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:54,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:54,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:54,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:54,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74c4c242[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:54,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:54,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,679 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:54,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:54,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:54,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:54,683 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,684 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/6770d72d283d4804a4e272e15f5eee86 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6770d72d283d4804a4e272e15f5eee86 2023-03-24 21:54:54,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:54,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:54,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:54,688 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:54,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:54,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:54,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:54,688 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:54,689 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 6770d72d283d4804a4e272e15f5eee86(size=12.2 K), total size for store is 24.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:54,689 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:54,689 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=12, startTime=1679694893834; duration=0sec 2023-03-24 21:54:54,689 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:54,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:54,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:54,696 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:54,696 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,797 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:54,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694954796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:54,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:54,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:54,800 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:54,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:54,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:54,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:54,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:54,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:54,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:54,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:54,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:54,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:54,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:54,806 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:54,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:54,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:54,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:54,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:54,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:54,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,809 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:54,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@223bfafb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:54,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:54,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:54,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:54,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:54,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:54,815 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,815 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:54,815 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,815 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:54,817 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:54,817 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:54,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:54,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:54,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:54,817 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:54,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:54,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:54,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:54,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:54,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:54,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,825 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:54,825 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:54,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,911 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/dd1f26e02aec40e896200f227688f457 2023-03-24 21:54:54,915 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/0b8a4f7a4d8249f0b5faa1fd94438a2c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c 2023-03-24 21:54:54,919 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 21:54:54,920 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/4673a95902824d8da87d03ff2adc7f08 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08 2023-03-24 21:54:54,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 21:54:54,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/dd1f26e02aec40e896200f227688f457 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457 2023-03-24 21:54:54,925 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:54,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:54,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 21:54:54,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:54,928 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for f67314e16f9680bd83bac15d2c4afabe in 492ms, sequenceid=254, compaction requested=true 2023-03-24 21:54:54,928 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:54,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:54,928 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:54,928 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:54,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:54,928 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:54,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:54,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:54,929 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:54,929 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,929 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:54,929 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:54,929 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:54,929 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:54,929 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,929 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c75d488ba7c94a7bbcdaa7a51d21a071, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=35.7 K 2023-03-24 21:54:54,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:54,930 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c75d488ba7c94a7bbcdaa7a51d21a071, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892607 2023-03-24 21:54:54,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:54,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d0a00886123740658fc341f420d2f57d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=35.7 K 2023-03-24 21:54:54,930 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d0ea9a1d51a84ecdaaab099b87801869, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679694892944 2023-03-24 21:54:54,931 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d0a00886123740658fc341f420d2f57d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892607 2023-03-24 21:54:54,931 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 0b8a4f7a4d8249f0b5faa1fd94438a2c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:54,931 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2c082ec855fc4cc4a3b0d03759853cb9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679694892944 2023-03-24 21:54:54,931 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:54,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:54,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:54,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:54,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:54,932 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4673a95902824d8da87d03ff2adc7f08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:54,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:54,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:54,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:54,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:54,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:54,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:54,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:54,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:54,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:54,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:54,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:54,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:54,934 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:54:54,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:54,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:54,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:54,935 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:54,944 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#250 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:54,944 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:55,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:55,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:55,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694955145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,252 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694955249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,352 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/9041d51d765349ee97f506ba04daf93c 2023-03-24 21:54:55,368 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/d7c6ba4009ec406a9332a3a3e18f375f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d7c6ba4009ec406a9332a3a3e18f375f 2023-03-24 21:54:55,369 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/b8b5f3d6b8df403895469e3ba1d87378 2023-03-24 21:54:55,379 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/d9534d3e166a4defb1423e698844dc25 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d9534d3e166a4defb1423e698844dc25 2023-03-24 21:54:55,385 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into d9534d3e166a4defb1423e698844dc25(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:55,385 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:55,385 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=13, startTime=1679694894928; duration=0sec 2023-03-24 21:54:55,385 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:55,385 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:55,387 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36586 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:55,387 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:55,387 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:55,387 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6770d72d283d4804a4e272e15f5eee86, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=35.7 K 2023-03-24 21:54:55,388 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6770d72d283d4804a4e272e15f5eee86, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1679694892607 2023-03-24 21:54:55,388 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b8d88e125b3745fcb4e5ba63efc55769, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1679694892944 2023-03-24 21:54:55,388 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting dd1f26e02aec40e896200f227688f457, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:55,392 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into d7c6ba4009ec406a9332a3a3e18f375f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:55,392 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:55,392 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=13, startTime=1679694894928; duration=0sec 2023-03-24 21:54:55,392 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:55,408 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:55,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694955453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694955681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,684 DEBUG [Thread-1604] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4126 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:55,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694955687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,689 DEBUG [Thread-1606] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4132 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:55,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694955695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,697 DEBUG [Thread-1600] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:55,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694955705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,710 DEBUG [Thread-1602] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4149 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:55,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694955755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,821 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/e4e4ff72183d4137b9c02e7a769b9095 2023-03-24 21:54:55,827 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/bed5684ab5aa4bbe90c342765b74bc5e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/bed5684ab5aa4bbe90c342765b74bc5e 2023-03-24 21:54:55,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/9041d51d765349ee97f506ba04daf93c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c 2023-03-24 21:54:55,838 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c, entries=150, sequenceid=270, filesize=11.9 K 2023-03-24 21:54:55,838 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into bed5684ab5aa4bbe90c342765b74bc5e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:55,838 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:55,838 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=13, startTime=1679694894928; duration=0sec 2023-03-24 21:54:55,838 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:55,839 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/b8b5f3d6b8df403895469e3ba1d87378 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378 2023-03-24 21:54:55,842 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378, entries=150, sequenceid=270, filesize=11.9 K 2023-03-24 21:54:55,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/e4e4ff72183d4137b9c02e7a769b9095 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095 2023-03-24 21:54:55,847 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095, entries=150, sequenceid=270, filesize=11.9 K 2023-03-24 21:54:55,847 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for f67314e16f9680bd83bac15d2c4afabe in 913ms, sequenceid=270, compaction requested=false 2023-03-24 21:54:55,847 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:55,847 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:55,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:55,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,851 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:55,851 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,851 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:55,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:55,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74430453[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:55,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:55,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:55,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:55,854 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:55,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:55,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:55,855 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,855 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:55,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:55,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:55,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:55,858 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:55,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:55,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:55,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:55,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:55,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:55,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:55,865 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:55,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:55,866 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:55,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:55,966 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:55,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:55,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:55,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:55,969 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:55,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:55,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:55,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:55,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:55,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,971 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:55,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:55,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:55,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:55,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:55,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:55,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:55,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:55,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:55,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:55,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:55,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:55,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:55,973 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:55,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:55,974 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:55,974 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:55,997 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/38b74507f2cd41688a487019fbb88d3a 2023-03-24 21:54:56,011 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/44ca4d4d4d8b4d85adfa64a9a2463632 2023-03-24 21:54:56,024 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/a1c5ce1171a940fbb8976b32cefd8562 2023-03-24 21:54:56,029 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/38b74507f2cd41688a487019fbb88d3a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a 2023-03-24 21:54:56,033 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a, entries=150, sequenceid=290, filesize=11.9 K 2023-03-24 21:54:56,033 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/44ca4d4d4d8b4d85adfa64a9a2463632 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632 2023-03-24 21:54:56,038 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632, entries=150, sequenceid=290, filesize=11.9 K 2023-03-24 21:54:56,038 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/a1c5ce1171a940fbb8976b32cefd8562 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562 2023-03-24 21:54:56,042 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562, entries=150, sequenceid=290, filesize=11.9 K 2023-03-24 21:54:56,042 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 69ms, sequenceid=290, compaction requested=true 2023-03-24 21:54:56,042 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:56,042 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,042 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,043 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,043 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,043 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,043 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc89ba1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,053 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,053 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,056 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,059 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,059 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,060 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,060 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,060 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:56,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,160 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,163 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,167 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:56,167 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:56,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,168 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,172 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,173 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,173 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49cc9c48[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,175 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,177 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,180 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,180 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,182 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,191 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,191 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:56,276 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:56,277 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:56,284 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/6671be2d2c5c4f7db83bce42b29b9a9d 2023-03-24 21:54:56,291 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,295 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,299 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:56,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,300 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:56,301 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,301 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@345d4ba8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,321 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,321 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694896320, max timestamp=1679694896332Num uniques 859; 2023-03-24 21:54:56,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:56,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694896320, max timestamp=1679694896332Num uniques 859; 2023-03-24 21:54:56,334 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:56,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:56,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694956367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,421 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,425 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:56,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:56,429 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58d0a0ec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,439 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,439 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,445 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,445 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:56,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:56,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694956471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,545 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,549 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,552 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,553 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,553 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,555 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:56,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,559 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,559 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:56,559 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,561 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,561 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79a2dc4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,565 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,566 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,571 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,581 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,581 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:56,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:56,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694956675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,681 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,690 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,693 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,693 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,693 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:56,693 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:56,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:56,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:56,695 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/2d26cada00884ec2aa72d36372e9c023 2023-03-24 21:54:56,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:56,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a19dfa3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:56,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,701 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:56,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:56,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:56,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:56,705 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:56,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:56,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:56,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:56,705 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:56,707 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,707 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:56,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,707 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9517fb8e3c194d53820a09e2bc4b891e 2023-03-24 21:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:56,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:56,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:56,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/6671be2d2c5c4f7db83bce42b29b9a9d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d 2023-03-24 21:54:56,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,714 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:56,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:56,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,714 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:56,715 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:56,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:56,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:56,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:56,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d, entries=150, sequenceid=301, filesize=11.9 K 2023-03-24 21:54:56,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/2d26cada00884ec2aa72d36372e9c023 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023 2023-03-24 21:54:56,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023, entries=150, sequenceid=301, filesize=11.9 K 2023-03-24 21:54:56,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9517fb8e3c194d53820a09e2bc4b891e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e 2023-03-24 21:54:56,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e, entries=150, sequenceid=301, filesize=11.9 K 2023-03-24 21:54:56,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for f67314e16f9680bd83bac15d2c4afabe in 450ms, sequenceid=301, compaction requested=true 2023-03-24 21:54:56,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:56,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:56,726 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:56,726 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:56,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:56,726 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:56,727 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:56,727 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:56,727 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:56,727 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,727 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:56,727 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d9534d3e166a4defb1423e698844dc25, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=48.0 K 2023-03-24 21:54:56,727 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,727 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d7c6ba4009ec406a9332a3a3e18f375f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=48.0 K 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d9534d3e166a4defb1423e698844dc25, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d7c6ba4009ec406a9332a3a3e18f375f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9041d51d765349ee97f506ba04daf93c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679694894470 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b8b5f3d6b8df403895469e3ba1d87378, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679694894470 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 38b74507f2cd41688a487019fbb88d3a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679694895132 2023-03-24 21:54:56,728 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 44ca4d4d4d8b4d85adfa64a9a2463632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679694895132 2023-03-24 21:54:56,729 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6671be2d2c5c4f7db83bce42b29b9a9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:56,729 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2d26cada00884ec2aa72d36372e9c023, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:56,738 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#261 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:56,738 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#262 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:56,746 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/bc07f0865c5a49de96bb4845c79c1f82 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/bc07f0865c5a49de96bb4845c79c1f82 2023-03-24 21:54:56,751 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into bc07f0865c5a49de96bb4845c79c1f82(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:56,751 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:56,751 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=12, startTime=1679694896726; duration=0sec 2023-03-24 21:54:56,751 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:56,751 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:54:56,752 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49174 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:54:56,752 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:56,752 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,752 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/bed5684ab5aa4bbe90c342765b74bc5e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=48.0 K 2023-03-24 21:54:56,752 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bed5684ab5aa4bbe90c342765b74bc5e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694894107 2023-03-24 21:54:56,753 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e4e4ff72183d4137b9c02e7a769b9095, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1679694894470 2023-03-24 21:54:56,753 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a1c5ce1171a940fbb8976b32cefd8562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679694895132 2023-03-24 21:54:56,753 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9517fb8e3c194d53820a09e2bc4b891e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:56,759 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:56,815 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:56,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:56,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:56,818 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:56,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:56,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:56,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:56,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:56,820 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:56,820 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:56,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:56,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:56,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:56,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:56,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:56,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:56,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:56,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:56,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:56,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:56,822 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=131.94 KB heapSize=383.31 KB 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:56,822 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:56,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:56,982 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:57,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694957008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:57,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694957111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,147 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/38a74fa4fa88417391a74f9705aa96bd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38a74fa4fa88417391a74f9705aa96bd 2023-03-24 21:54:57,152 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into 38a74fa4fa88417391a74f9705aa96bd(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:57,152 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:57,152 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=12, startTime=1679694896726; duration=0sec 2023-03-24 21:54:57,152 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:57,171 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/45bffbfc50884f25b2cc41375c3bbf30 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/45bffbfc50884f25b2cc41375c3bbf30 2023-03-24 21:54:57,175 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 45bffbfc50884f25b2cc41375c3bbf30(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:57,175 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:57,175 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=12, startTime=1679694896726; duration=0sec 2023-03-24 21:54:57,175 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:57,267 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/d04348831c2545048d7c4b47bae5671f 2023-03-24 21:54:57,278 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/908956fb2b654ee8b38d94e4cefa19c1 2023-03-24 21:54:57,288 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=64.85 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/686da21ce349457ca950ef376d5795ce 2023-03-24 21:54:57,292 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/d04348831c2545048d7c4b47bae5671f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f 2023-03-24 21:54:57,296 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f, entries=150, sequenceid=334, filesize=11.9 K 2023-03-24 21:54:57,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/908956fb2b654ee8b38d94e4cefa19c1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1 2023-03-24 21:54:57,302 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1, entries=150, sequenceid=334, filesize=11.9 K 2023-03-24 21:54:57,303 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/686da21ce349457ca950ef376d5795ce as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce 2023-03-24 21:54:57,306 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce, entries=150, sequenceid=334, filesize=11.9 K 2023-03-24 21:54:57,307 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~383.27 KB/392464, currentSize=46.96 KB/48090 for f67314e16f9680bd83bac15d2c4afabe in 486ms, sequenceid=334, compaction requested=false 2023-03-24 21:54:57,307 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:57,307 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7447c143[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:57,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:57,319 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:57,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,323 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,327 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,327 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,337 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,337 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694897353, max timestamp=1679694897370Num uniques 859; 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694897353, max timestamp=1679694897370Num uniques 859; 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694897353, max timestamp=1679694897370Num uniques 859; 2023-03-24 21:54:57,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:57,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:57,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694957422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,437 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:57,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:57,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:57,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:57,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:57,441 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:57,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:57,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:57,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:57,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:57,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,448 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,449 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@294dc8e7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,459 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,459 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,467 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,470 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,471 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,471 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,471 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,471 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:57,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694957525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,571 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:57,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:57,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:57,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:57,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:57,575 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,577 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:57,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:57,579 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,579 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:57,579 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,579 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f6f3407[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,586 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,589 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,589 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:57,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,595 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,595 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,595 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,695 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:57,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:57,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:57,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:57,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:57,699 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:57,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:57,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:57,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,702 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:57,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:57,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:57,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:57,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:57,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:57,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bef088e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,713 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,713 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:57,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,721 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,721 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:57,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694957727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/c5d1ef5939eb4950a60303dbd1b7c672 2023-03-24 21:54:57,821 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:57,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:57,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:57,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:57,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:57,824 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:57,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:57,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:57,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:57,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:57,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:57,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:57,828 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,828 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:57,828 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,828 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51688f69[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,834 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,838 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,838 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,849 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,849 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,950 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:57,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:57,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:57,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:57,953 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:57,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:57,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:57,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,956 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:57,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:57,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:57,957 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,957 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:57,957 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:57,957 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:57,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:57,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,959 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,960 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,960 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:57,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16257f53[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:57,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:57,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:57,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:57,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:57,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:57,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:57,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:57,967 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:57,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:57,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:57,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:57,968 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:57,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:57,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,970 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:57,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:57,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:57,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:57,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:57,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:57,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:57,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:57,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:57,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:57,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,975 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:57,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:57,975 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:57,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:57,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:57,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:58,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:58,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694958032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,075 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:58,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:58,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:58,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:58,079 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:58,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:58,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:58,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:58,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:58,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:58,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:58,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,084 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:58,084 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:58,084 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:58,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:58,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:58,085 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,085 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:58,085 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:58,085 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:58,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:58,087 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:58,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:58,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106cbe9c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:58,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:58,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:58,099 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:58,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:58,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:58,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:58,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:58,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:58,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:58,104 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:58,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:58,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:58,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:58,104 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:58,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:58,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:58,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:58,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:58,119 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:58,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:58,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:58,120 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:58,145 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/67467e11a31c4a15a8d50816ac9e2f54 2023-03-24 21:54:58,174 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/670d520e2be6460b8562214e2f5a6cf7 2023-03-24 21:54:58,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/c5d1ef5939eb4950a60303dbd1b7c672 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672 2023-03-24 21:54:58,182 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 21:54:58,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/67467e11a31c4a15a8d50816ac9e2f54 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54 2023-03-24 21:54:58,186 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 21:54:58,187 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/670d520e2be6460b8562214e2f5a6cf7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7 2023-03-24 21:54:58,191 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7, entries=150, sequenceid=347, filesize=11.9 K 2023-03-24 21:54:58,192 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f67314e16f9680bd83bac15d2c4afabe in 874ms, sequenceid=347, compaction requested=true 2023-03-24 21:54:58,192 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:58,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:58,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:58,192 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:58,192 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:58,192 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:58,193 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:58,193 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:58,193 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:54:58,194 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:54:58,194 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,194 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38a74fa4fa88417391a74f9705aa96bd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=36.4 K 2023-03-24 21:54:58,194 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,194 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/bc07f0865c5a49de96bb4845c79c1f82, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=36.4 K 2023-03-24 21:54:58,194 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 38a74fa4fa88417391a74f9705aa96bd, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:58,194 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bc07f0865c5a49de96bb4845c79c1f82, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:58,194 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d04348831c2545048d7c4b47bae5671f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679694896359 2023-03-24 21:54:58,195 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 908956fb2b654ee8b38d94e4cefa19c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679694896359 2023-03-24 21:54:58,195 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c5d1ef5939eb4950a60303dbd1b7c672, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:54:58,195 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 67467e11a31c4a15a8d50816ac9e2f54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:54:58,209 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#270 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:58,211 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#271 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:58,220 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:58,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:58,235 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:58,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:58,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:58,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,237 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:58,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:58,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:58,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:58,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:58,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:58,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:58,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:58,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:58,244 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 21:54:58,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:58,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:58,244 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:58,245 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:58,245 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:58,245 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:54:58,248 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/e357acc6a96a4ada8f6cb4fb7a9b5739 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/e357acc6a96a4ada8f6cb4fb7a9b5739 2023-03-24 21:54:58,252 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/e49fe5519ef34fb691d2c4512008b6c4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e49fe5519ef34fb691d2c4512008b6c4 2023-03-24 21:54:58,253 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into e357acc6a96a4ada8f6cb4fb7a9b5739(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:58,253 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:58,253 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=13, startTime=1679694898192; duration=0sec 2023-03-24 21:54:58,254 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:54:58,254 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:54:58,255 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:54:58,255 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:54:58,255 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,256 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/45bffbfc50884f25b2cc41375c3bbf30, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=36.4 K 2023-03-24 21:54:58,256 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 45bffbfc50884f25b2cc41375c3bbf30, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679694896266 2023-03-24 21:54:58,256 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 686da21ce349457ca950ef376d5795ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1679694896359 2023-03-24 21:54:58,256 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 670d520e2be6460b8562214e2f5a6cf7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:54:58,259 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into e49fe5519ef34fb691d2c4512008b6c4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:58,259 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:58,259 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=13, startTime=1679694898192; duration=0sec 2023-03-24 21:54:58,259 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:58,261 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/860ab4f1a84842e0bb72cd1e7ac9f5d5 2023-03-24 21:54:58,268 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:54:58,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:58,536 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:58,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:58,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694958557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:58,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694958660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,674 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/29060c41eedb4538a29c25e15b6ba871 2023-03-24 21:54:58,681 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/7602e2021366477aa9d5ee97605fc69e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7602e2021366477aa9d5ee97605fc69e 2023-03-24 21:54:58,689 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/4d46b165fea64f9b852580ee1a706710 2023-03-24 21:54:58,692 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 7602e2021366477aa9d5ee97605fc69e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:54:58,692 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:58,692 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=13, startTime=1679694898192; duration=0sec 2023-03-24 21:54:58,693 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:54:58,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/860ab4f1a84842e0bb72cd1e7ac9f5d5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5 2023-03-24 21:54:58,698 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5, entries=150, sequenceid=383, filesize=11.9 K 2023-03-24 21:54:58,698 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/29060c41eedb4538a29c25e15b6ba871 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871 2023-03-24 21:54:58,702 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871, entries=150, sequenceid=383, filesize=11.9 K 2023-03-24 21:54:58,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/4d46b165fea64f9b852580ee1a706710 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710 2023-03-24 21:54:58,709 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710, entries=150, sequenceid=383, filesize=11.9 K 2023-03-24 21:54:58,710 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for f67314e16f9680bd83bac15d2c4afabe in 466ms, sequenceid=383, compaction requested=false 2023-03-24 21:54:58,710 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:54:58,710 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:54:58,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:54:58,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:54:58,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:54:58,712 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,712 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:54:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:54:58,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73247f67[Count = 0] remaining members to acquire global barrier 2023-03-24 21:54:58,714 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:54:58,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:54:58,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:54:58,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:54:58,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:54:58,717 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:58,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:54:58,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:54:58,720 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:54:58,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:54:58,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:54:58,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:54:58,720 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:54:58,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:54:58,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:58,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:54:58,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:54:58,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:58,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:54:58,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:58,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:54:58,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:54:58,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:54:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:54:58,735 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:54:58,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:54:58,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:54:58,736 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:54:58,736 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:54:58,836 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:54:58,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:54:58,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:54:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:54:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:54:58,839 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:54:58,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:54:58,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:54:58,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,841 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:54:58,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:54:58,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:54:58,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:58,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:54:58,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:54:58,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:54:58,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:54:58,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:54:58,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:54:58,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:54:58,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:54:58,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:54:58,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:54:58,843 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:54:58,843 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:58,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:54:58,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:54:58,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:54:58,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694898906, max timestamp=1679694898920Num uniques 859; 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694898906, max timestamp=1679694898920Num uniques 859; 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694898906, max timestamp=1679694898920Num uniques 859; 2023-03-24 21:54:58,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:54:58,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:58,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694958970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694959073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,251 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/b3be4938660b4dbaa3cbf77fc0b4cc68 2023-03-24 21:54:59,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694959277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694959581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,668 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/dfa959328a854b588b4bd27339d1f814 2023-03-24 21:54:59,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52642 deadline: 1679694959696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,701 DEBUG [Thread-1604] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:59,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52622 deadline: 1679694959708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,710 DEBUG [Thread-1600] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:59,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52580 deadline: 1679694959721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,724 DEBUG [Thread-1606] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:54:59,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:54:59,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:52592 deadline: 1679694959750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:54:59,753 DEBUG [Thread-1602] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:00,078 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=397 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/62ad9f5123ad4aef93a05d49d8bbea2e 2023-03-24 21:55:00,082 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/b3be4938660b4dbaa3cbf77fc0b4cc68 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68 2023-03-24 21:55:00,086 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68, entries=150, sequenceid=397, filesize=11.9 K 2023-03-24 21:55:00,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/dfa959328a854b588b4bd27339d1f814 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814 2023-03-24 21:55:00,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694960085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,091 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814, entries=150, sequenceid=397, filesize=11.9 K 2023-03-24 21:55:00,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/62ad9f5123ad4aef93a05d49d8bbea2e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e 2023-03-24 21:55:00,101 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e, entries=150, sequenceid=397, filesize=11.9 K 2023-03-24 21:55:00,101 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f67314e16f9680bd83bac15d2c4afabe in 1258ms, sequenceid=397, compaction requested=true 2023-03-24 21:55:00,101 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:00,102 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:00,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:00,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:00,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:00,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,104 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:00,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:00,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6622a5ad[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:00,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:00,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:00,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:00,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:00,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:00,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:00,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:00,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:00,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:00,110 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:00,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:00,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:00,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:00,110 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:00,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:00,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:00,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:00,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,118 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:00,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:00,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:00,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,121 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:00,121 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:00,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:00,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:00,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:00,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:00,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:00,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:00,221 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:00,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:00,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:00,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:00,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:00,224 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:00,224 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:00,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:00,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:00,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:00,226 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:00,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:00,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:00,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:00,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:00,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:00,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:00,227 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:00,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:00,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:00,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:00,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:00,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:00,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:00,228 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:00,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:00,229 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:00,636 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/fb93b5a5a3224da994d2aeb58eac68c4 2023-03-24 21:55:01,047 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/da53f5e207b840b3bdbf018946025d2c 2023-03-24 21:55:01,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:01,096 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:01,127 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:01,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694961124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:01,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694961229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:01,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694961432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,459 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=433 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/7b5129a56bd0478e9e29ade19d144efa 2023-03-24 21:55:01,463 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/fb93b5a5a3224da994d2aeb58eac68c4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4 2023-03-24 21:55:01,468 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4, entries=150, sequenceid=433, filesize=11.9 K 2023-03-24 21:55:01,469 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/da53f5e207b840b3bdbf018946025d2c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c 2023-03-24 21:55:01,472 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c, entries=150, sequenceid=433, filesize=11.9 K 2023-03-24 21:55:01,473 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/7b5129a56bd0478e9e29ade19d144efa as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa 2023-03-24 21:55:01,477 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa, entries=150, sequenceid=433, filesize=11.9 K 2023-03-24 21:55:01,477 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for f67314e16f9680bd83bac15d2c4afabe in 1249ms, sequenceid=433, compaction requested=true 2023-03-24 21:55:01,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:01,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:01,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:01,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:01,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,478 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:01,478 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:01,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:01,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:01,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:01,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:01,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bc091dd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:01,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:01,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:01,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:01,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:01,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:01,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:01,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:01,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:01,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:01,486 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:01,487 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:01,487 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:01,487 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:01,487 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:01,489 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,489 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:01,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,489 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:01,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:01,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:01,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:01,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:01,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:01,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:01,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:01,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:01,493 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:01,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:01,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:01,494 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:01,494 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,494 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:01,494 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,494 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:01,494 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:01,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:01,594 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:01,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:01,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:01,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:01,598 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:01,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:01,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:01,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:01,599 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,599 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:01,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:01,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:01,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:01,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:01,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:01,601 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:01,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:01,601 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:01,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:01,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:01,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:01,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:01,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:01,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:01,602 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:01,602 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:01,602 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:01,602 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:01,602 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:01,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:01,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:01,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:01,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:01,603 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:01,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:01,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694901794, max timestamp=1679694901803Num uniques 859; 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694901794, max timestamp=1679694901803Num uniques 859; 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694901794, max timestamp=1679694901803Num uniques 859; 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:01,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:01,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:01,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694961870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:01,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:01,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694961975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,017 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/13c4bddc0e92496898058d3af09a1cc3 2023-03-24 21:55:02,028 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/0669676b7a4e4e60a3a2b0003acc3e02 2023-03-24 21:55:02,037 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=444 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/6a4a930d68cc4754bd03faea747e53e8 2023-03-24 21:55:02,041 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/13c4bddc0e92496898058d3af09a1cc3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3 2023-03-24 21:55:02,045 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3, entries=150, sequenceid=444, filesize=11.9 K 2023-03-24 21:55:02,046 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/0669676b7a4e4e60a3a2b0003acc3e02 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02 2023-03-24 21:55:02,049 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02, entries=150, sequenceid=444, filesize=11.9 K 2023-03-24 21:55:02,050 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/6a4a930d68cc4754bd03faea747e53e8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8 2023-03-24 21:55:02,054 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8, entries=150, sequenceid=444, filesize=11.9 K 2023-03-24 21:55:02,055 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for f67314e16f9680bd83bac15d2c4afabe in 453ms, sequenceid=444, compaction requested=true 2023-03-24 21:55:02,055 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:02,055 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,057 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,057 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,057 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,057 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ead5c67[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,059 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,059 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,063 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,065 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,065 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,067 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,067 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,067 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,067 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,071 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,071 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,172 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,175 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:02,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,182 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:55:02,182 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:02,182 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:02,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:02,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:02,183 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:02,183 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:02,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,188 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e22d192[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,192 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,192 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,194 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,194 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,194 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,197 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,197 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,199 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,199 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,199 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,202 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,205 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,205 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,205 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,206 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,215 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:02,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 384 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694962213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,306 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,309 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d5e0c36[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,320 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:02,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 386 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694962317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,323 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,324 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,325 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,325 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,325 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,332 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,332 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,332 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,332 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:02,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,432 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,437 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,440 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,440 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,440 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,441 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,441 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524f2a39[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,448 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,450 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,451 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:02,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,460 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,460 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:02,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 388 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694962522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,560 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,568 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,568 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,568 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,568 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,568 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16c51a73[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,586 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,586 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,593 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,593 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,597 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/510c2c8e85a94359ac57a30d8c9f48de 2023-03-24 21:55:02,694 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,697 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,701 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,701 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,702 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,702 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,702 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,702 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e83f778[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,713 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,713 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,719 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,719 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:02,720 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,719 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,720 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,820 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,823 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,826 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:02,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 390 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694962825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,828 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,830 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,832 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c16b22[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,835 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,835 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,842 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,842 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,843 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,854 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:02,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,854 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,854 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,954 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:02,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:02,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:02,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:02,958 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:02,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:02,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:02,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,960 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:02,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:02,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:02,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:02,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:02,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:02,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:02,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:02,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,961 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:02,962 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:02,962 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:02,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:02,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:02,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:02,962 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:02,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:02,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28748781[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:02,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:02,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,968 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:02,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:02,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:02,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:02,969 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:02,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:02,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:02,972 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:02,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:02,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:02,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:02,972 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:02,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:02,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:02,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:02,981 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:02,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:02,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:02,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:02,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:02,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:02,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:02,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:02,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:02,985 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:02,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:02,985 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:02,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:02,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,009 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/ccd5a15227f14b5ba512c57444b97186 2023-03-24 21:55:03,085 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:03,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:03,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:03,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:03,088 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:03,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:03,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:03,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:03,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:03,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:03,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:03,092 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:03,092 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:03,094 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:03,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@165c167e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:03,096 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:03,096 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,098 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:03,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:03,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:03,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:03,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:03,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,099 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:03,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:03,102 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:03,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:03,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:03,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:03,102 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:03,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:03,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:03,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,108 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:03,108 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:03,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,209 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:03,211 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:03,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:03,212 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:03,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:03,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:03,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,214 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:03,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:03,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,215 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:03,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:03,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:03,216 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,216 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:03,216 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:03,216 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:03,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:03,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,218 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:03,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77851bfa[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:03,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:03,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,221 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,221 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:03,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:03,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:03,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:03,223 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:03,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:03,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:03,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:03,225 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:03,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:03,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:03,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:03,225 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:03,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,228 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,229 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,232 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,232 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:03,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:03,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,232 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,232 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,232 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:03,232 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:03,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,235 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:03,331 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:03,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 392 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694963329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,332 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:03,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:03,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:03,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:03,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:03,336 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:03,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:03,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:03,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,338 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:03,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:03,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:03,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:03,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:03,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:03,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:03,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:03,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,343 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:03,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:03,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f95e653[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:03,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:03,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,348 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:03,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:03,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:03,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:03,350 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:03,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:03,353 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:03,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:03,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:03,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:03,353 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:03,355 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,355 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,355 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,364 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:03,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,365 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:03,364 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,365 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,365 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,365 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:03,365 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:03,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:03,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9a6778ce97bc4870996d40891a668d7b 2023-03-24 21:55:03,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/510c2c8e85a94359ac57a30d8c9f48de as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de 2023-03-24 21:55:03,427 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de, entries=150, sequenceid=481, filesize=11.9 K 2023-03-24 21:55:03,428 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/ccd5a15227f14b5ba512c57444b97186 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186 2023-03-24 21:55:03,432 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186, entries=150, sequenceid=481, filesize=11.9 K 2023-03-24 21:55:03,432 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9a6778ce97bc4870996d40891a668d7b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b 2023-03-24 21:55:03,435 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b, entries=150, sequenceid=481, filesize=11.9 K 2023-03-24 21:55:03,436 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for f67314e16f9680bd83bac15d2c4afabe in 1254ms, sequenceid=481, compaction requested=true 2023-03-24 21:55:03,436 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:03,436 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:03,436 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 21:55:03,436 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 21:55:03,438 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73934 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 21:55:03,438 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73934 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 21:55:03,438 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:55:03,439 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,439 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/e357acc6a96a4ada8f6cb4fb7a9b5739, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=72.2 K 2023-03-24 21:55:03,439 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:55:03,439 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,439 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e49fe5519ef34fb691d2c4512008b6c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=72.2 K 2023-03-24 21:55:03,439 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e357acc6a96a4ada8f6cb4fb7a9b5739, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:55:03,439 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e49fe5519ef34fb691d2c4512008b6c4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:55:03,440 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 860ab4f1a84842e0bb72cd1e7ac9f5d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679694897411 2023-03-24 21:55:03,440 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 29060c41eedb4538a29c25e15b6ba871, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679694897411 2023-03-24 21:55:03,440 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b3be4938660b4dbaa3cbf77fc0b4cc68, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679694898547 2023-03-24 21:55:03,440 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting dfa959328a854b588b4bd27339d1f814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679694898547 2023-03-24 21:55:03,440 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting fb93b5a5a3224da994d2aeb58eac68c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1679694898963 2023-03-24 21:55:03,441 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting da53f5e207b840b3bdbf018946025d2c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1679694898963 2023-03-24 21:55:03,441 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 13c4bddc0e92496898058d3af09a1cc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679694901106 2023-03-24 21:55:03,441 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0669676b7a4e4e60a3a2b0003acc3e02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679694901106 2023-03-24 21:55:03,441 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 510c2c8e85a94359ac57a30d8c9f48de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:03,442 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ccd5a15227f14b5ba512c57444b97186, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:03,465 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#288 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:03,465 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:03,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:03,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:03,469 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#289 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:03,471 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:03,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:03,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:03,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:03,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:03,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:03,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:03,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:03,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:03,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:03,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:03,482 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:03,482 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:03,490 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/a03020d03710436fbd3b9d212d1f8a7b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a03020d03710436fbd3b9d212d1f8a7b 2023-03-24 21:55:03,495 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into a03020d03710436fbd3b9d212d1f8a7b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:03,495 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:03,495 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=10, startTime=1679694903436; duration=0sec 2023-03-24 21:55:03,495 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:03,495 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 21:55:03,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73934 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 21:55:03,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:55:03,497 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,497 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7602e2021366477aa9d5ee97605fc69e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=72.2 K 2023-03-24 21:55:03,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7602e2021366477aa9d5ee97605fc69e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1679694896992 2023-03-24 21:55:03,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4d46b165fea64f9b852580ee1a706710, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=383, earliestPutTs=1679694897411 2023-03-24 21:55:03,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 62ad9f5123ad4aef93a05d49d8bbea2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=397, earliestPutTs=1679694898547 2023-03-24 21:55:03,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7b5129a56bd0478e9e29ade19d144efa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=433, earliestPutTs=1679694898963 2023-03-24 21:55:03,499 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6a4a930d68cc4754bd03faea747e53e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=444, earliestPutTs=1679694901106 2023-03-24 21:55:03,499 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9a6778ce97bc4870996d40891a668d7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:03,509 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#291 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:03,888 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/5a834ac5c1e5472ca2845796e26a57d8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/5a834ac5c1e5472ca2845796e26a57d8 2023-03-24 21:55:03,891 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/06659d264f6e4be3af0c8f812d0dbf58 2023-03-24 21:55:03,893 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into 5a834ac5c1e5472ca2845796e26a57d8(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:03,893 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:03,893 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=10, startTime=1679694903436; duration=0sec 2023-03-24 21:55:03,893 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:03,901 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf 2023-03-24 21:55:03,909 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=491 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/d3917adb5be94e1ea352eb86c7f53e80 2023-03-24 21:55:03,913 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/06659d264f6e4be3af0c8f812d0dbf58 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58 2023-03-24 21:55:03,920 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58, entries=100, sequenceid=491, filesize=9.5 K 2023-03-24 21:55:03,921 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf 2023-03-24 21:55:03,922 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/6466fa6a8d0c43fd8f4188c47bd4ae40 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6466fa6a8d0c43fd8f4188c47bd4ae40 2023-03-24 21:55:03,925 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf, entries=100, sequenceid=491, filesize=9.5 K 2023-03-24 21:55:03,925 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 6466fa6a8d0c43fd8f4188c47bd4ae40(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:03,926 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:03,926 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/d3917adb5be94e1ea352eb86c7f53e80 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80 2023-03-24 21:55:03,926 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=10, startTime=1679694903436; duration=0sec 2023-03-24 21:55:03,926 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:03,929 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80, entries=100, sequenceid=491, filesize=9.5 K 2023-03-24 21:55:03,930 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 448ms, sequenceid=491, compaction requested=false 2023-03-24 21:55:03,930 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:03,930 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:03,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:03,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:03,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:03,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:03,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:03,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:03,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73512eaf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:03,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:03,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,936 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:03,936 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:03,936 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:03,936 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,936 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:03,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,938 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:03,938 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,938 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:03,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:03,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:03,941 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:03,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:03,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:03,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:03,941 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:03,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:03,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:03,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:03,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:03,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:03,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:03,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:03,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,948 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:03,949 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:03,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:03,949 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:03,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:03,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,049 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,053 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,055 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,055 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,056 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,056 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,058 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,058 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,058 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,058 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,058 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,058 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:04,058 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,059 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,061 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@548395d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,064 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,066 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,066 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,066 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,066 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,069 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,069 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,073 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,075 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,076 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,076 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,076 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,176 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,181 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,182 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,183 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,183 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,183 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,184 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,184 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,184 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,184 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,184 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,184 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:04,185 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,186 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,186 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,186 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,186 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5acda8c5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,192 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,195 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,195 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,204 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,204 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,204 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,205 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,305 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,308 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,315 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,315 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,315 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:04,315 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aba6ab1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,328 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,328 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,328 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,331 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,331 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,333 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,333 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,333 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,334 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,338 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,338 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,338 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:04,351 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:04,351 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694904384, max timestamp=1679694904408Num uniques 859; 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694904384, max timestamp=1679694904408Num uniques 859; 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694904384, max timestamp=1679694904408Num uniques 859; 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:04,410 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:04,439 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,442 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,447 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:04,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70e4191f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,461 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,462 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,463 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,463 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,464 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:04,470 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,470 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:04,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694964481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,570 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,573 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,577 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,577 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,577 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:04,577 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3422448f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:04,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694964586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,594 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,597 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,597 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,597 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,597 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,597 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,697 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:04,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:04,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:04,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:04,701 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:04,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:04,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:04,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:04,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:04,713 DEBUG [Thread-1613] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f775631 to 127.0.0.1:57724 2023-03-24 21:55:04,713 DEBUG [Thread-1613] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:04,714 DEBUG [Thread-1615] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x156e01b7 to 127.0.0.1:57724 2023-03-24 21:55:04,714 DEBUG [Thread-1615] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:04,716 DEBUG [Thread-1617] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c004e36 to 127.0.0.1:57724 2023-03-24 21:55:04,716 DEBUG [Thread-1617] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:04,717 DEBUG [Thread-1619] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d47b4b3 to 127.0.0.1:57724 2023-03-24 21:55:04,717 DEBUG [Thread-1619] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:04,717 DEBUG [Thread-1611] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a7e7bef to 127.0.0.1:57724 2023-03-24 21:55:04,718 DEBUG [Thread-1611] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:04,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:04,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:04,723 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,723 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. started... 2023-03-24 21:55:04,723 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. as already flushing 2023-03-24 21:55:04,723 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:04,723 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,724 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:04,724 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:04,726 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,726 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:04,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e494e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:04,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:04,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,730 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:04,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:04,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:04,730 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:04,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:04,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:04,735 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:04,735 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:04,735 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:04,735 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:04,735 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:04,737 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,737 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,737 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:04,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:04,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:04,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:04,742 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:04,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:04,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:04,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:04,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:04,742 DEBUG [Thread-1610] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:04,742 DEBUG [Thread-1610] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:04,759 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/88a804c09ca0443eb3c7d03118a82efe 2023-03-24 21:55:04,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:04,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 439 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694964789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:04,842 DEBUG [Thread-1610] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:04,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:05,092 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:05,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 441 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:52652 deadline: 1679694965092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:05,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/0d8375ff02d44f1f991bbfe4ea77e60d 2023-03-24 21:55:05,578 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=507 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8cda9242e36942d69f89b8a1b846ab01 2023-03-24 21:55:05,581 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/88a804c09ca0443eb3c7d03118a82efe as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe 2023-03-24 21:55:05,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe, entries=200, sequenceid=507, filesize=14.3 K 2023-03-24 21:55:05,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/0d8375ff02d44f1f991bbfe4ea77e60d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d 2023-03-24 21:55:05,588 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d, entries=150, sequenceid=507, filesize=11.9 K 2023-03-24 21:55:05,589 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8cda9242e36942d69f89b8a1b846ab01 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01 2023-03-24 21:55:05,592 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01, entries=150, sequenceid=507, filesize=11.9 K 2023-03-24 21:55:05,592 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for f67314e16f9680bd83bac15d2c4afabe in 1241ms, sequenceid=507, compaction requested=true 2023-03-24 21:55:05,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:05,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:05,592 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:05,592 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:05,592 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:05,593 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:05,593 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37576 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:05,593 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/A is initiating minor compaction (all files) 2023-03-24 21:55:05,593 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:05,593 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/A in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:05,593 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/B is initiating minor compaction (all files) 2023-03-24 21:55:05,593 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/5a834ac5c1e5472ca2845796e26a57d8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=36.7 K 2023-03-24 21:55:05,593 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/B in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:05,593 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a03020d03710436fbd3b9d212d1f8a7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.3 K 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5a834ac5c1e5472ca2845796e26a57d8, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a03020d03710436fbd3b9d212d1f8a7b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 06659d264f6e4be3af0c8f812d0dbf58, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679694902208 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3fff4b6dbe6e4c4fbbafa4bdf980fcdf, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679694902208 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 88a804c09ca0443eb3c7d03118a82efe, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1679694904340 2023-03-24 21:55:05,594 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0d8375ff02d44f1f991bbfe4ea77e60d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1679694904346 2023-03-24 21:55:05,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:05,596 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:05,596 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:05,597 DEBUG [Thread-1608] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7875eca1 to 127.0.0.1:57724 2023-03-24 21:55:05,597 DEBUG [Thread-1608] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:05,601 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#B#compaction#297 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:05,602 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#A#compaction#298 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:06,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/df9a45042ad643468b751c4785e9b00f 2023-03-24 21:55:06,012 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/24f2b8ad5d204ea08d73748b6e7eb58c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/24f2b8ad5d204ea08d73748b6e7eb58c 2023-03-24 21:55:06,012 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/f08a952e70b9450e97089e1465c49cb9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f08a952e70b9450e97089e1465c49cb9 2023-03-24 21:55:06,016 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/B of f67314e16f9680bd83bac15d2c4afabe into f08a952e70b9450e97089e1465c49cb9(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:06,016 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/A of f67314e16f9680bd83bac15d2c4afabe into 24f2b8ad5d204ea08d73748b6e7eb58c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:06,016 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:06,016 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:06,016 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/B, priority=13, startTime=1679694905592; duration=0sec 2023-03-24 21:55:06,017 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/A, priority=13, startTime=1679694905592; duration=0sec 2023-03-24 21:55:06,017 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:06,017 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:06,017 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:06,018 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:06,018 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): f67314e16f9680bd83bac15d2c4afabe/C is initiating minor compaction (all files) 2023-03-24 21:55:06,018 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of f67314e16f9680bd83bac15d2c4afabe/C in TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:06,018 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6466fa6a8d0c43fd8f4188c47bd4ae40, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp, totalSize=34.3 K 2023-03-24 21:55:06,018 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6466fa6a8d0c43fd8f4188c47bd4ae40, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1679694901845 2023-03-24 21:55:06,018 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d3917adb5be94e1ea352eb86c7f53e80, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=491, earliestPutTs=1679694902208 2023-03-24 21:55:06,019 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8cda9242e36942d69f89b8a1b846ab01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=507, earliestPutTs=1679694904346 2023-03-24 21:55:06,024 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): f67314e16f9680bd83bac15d2c4afabe#C#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:06,416 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/4877d23f7ff54fe6b2bfb910fddfe1f7 2023-03-24 21:55:06,436 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/186bc5c99455404db9af63a6b03d85f7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/186bc5c99455404db9af63a6b03d85f7 2023-03-24 21:55:06,439 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in f67314e16f9680bd83bac15d2c4afabe/C of f67314e16f9680bd83bac15d2c4afabe into 186bc5c99455404db9af63a6b03d85f7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:06,439 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:06,439 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe., storeName=f67314e16f9680bd83bac15d2c4afabe/C, priority=13, startTime=1679694905592; duration=0sec 2023-03-24 21:55:06,439 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:06,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=545 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8f7e6206877d462ea483fcfa169a9d66 2023-03-24 21:55:06,829 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/df9a45042ad643468b751c4785e9b00f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df9a45042ad643468b751c4785e9b00f 2023-03-24 21:55:06,832 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df9a45042ad643468b751c4785e9b00f, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 21:55:06,833 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/4877d23f7ff54fe6b2bfb910fddfe1f7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4877d23f7ff54fe6b2bfb910fddfe1f7 2023-03-24 21:55:06,836 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4877d23f7ff54fe6b2bfb910fddfe1f7, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 21:55:06,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/8f7e6206877d462ea483fcfa169a9d66 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8f7e6206877d462ea483fcfa169a9d66 2023-03-24 21:55:06,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8f7e6206877d462ea483fcfa169a9d66, entries=150, sequenceid=545, filesize=11.9 K 2023-03-24 21:55:06,840 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 1244ms, sequenceid=545, compaction requested=false 2023-03-24 21:55:06,841 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:09,723 DEBUG [Thread-1604] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x018daa76 to 127.0.0.1:57724 2023-03-24 21:55:09,724 DEBUG [Thread-1604] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:09,736 DEBUG [Thread-1600] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18111197 to 127.0.0.1:57724 2023-03-24 21:55:09,736 DEBUG [Thread-1600] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:09,790 DEBUG [Thread-1606] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c25b245 to 127.0.0.1:57724 2023-03-24 21:55:09,790 DEBUG [Thread-1606] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:09,802 DEBUG [Thread-1602] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a1b30ad to 127.0.0.1:57724 2023-03-24 21:55:09,802 DEBUG [Thread-1602] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:09,802 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:55:09,802 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-24 21:55:09,802 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 25 2023-03-24 21:55:09,802 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 32 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 312 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 5922 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 5787 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 5975 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 5997 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 5799 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:55:09,803 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:55:09,803 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1594c869 to 127.0.0.1:57724 2023-03-24 21:55:09,803 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:09,804 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:55:09,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:55:09,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:09,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:09,807 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694909807"}]},"ts":"1679694909807"} 2023-03-24 21:55:09,808 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:55:09,810 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:55:09,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, UNASSIGN}] 2023-03-24 21:55:09,812 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, UNASSIGN 2023-03-24 21:55:09,813 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f67314e16f9680bd83bac15d2c4afabe, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:09,813 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694909813"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694909813"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694909813"}]},"ts":"1679694909813"} 2023-03-24 21:55:09,814 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:09,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:09,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing f67314e16f9680bd83bac15d2c4afabe, disabling compactions & flushes 2023-03-24 21:55:09,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. after waiting 0 ms 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:09,966 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing f67314e16f9680bd83bac15d2c4afabe 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=A 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=B 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK f67314e16f9680bd83bac15d2c4afabe, store=C 2023-03-24 21:55:09,966 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:10,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:10,372 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/331bddb069c84b25b4277d7b7c69e382 2023-03-24 21:55:10,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:10,781 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/8139121778ad48f08048e730cafa182e 2023-03-24 21:55:10,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:11,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=555 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9f93ec7cd69d4cf5bca6800523d1aafc 2023-03-24 21:55:11,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/A/331bddb069c84b25b4277d7b7c69e382 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/331bddb069c84b25b4277d7b7c69e382 2023-03-24 21:55:11,196 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/331bddb069c84b25b4277d7b7c69e382, entries=100, sequenceid=555, filesize=9.5 K 2023-03-24 21:55:11,197 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/B/8139121778ad48f08048e730cafa182e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/8139121778ad48f08048e730cafa182e 2023-03-24 21:55:11,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/8139121778ad48f08048e730cafa182e, entries=100, sequenceid=555, filesize=9.5 K 2023-03-24 21:55:11,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/.tmp/C/9f93ec7cd69d4cf5bca6800523d1aafc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9f93ec7cd69d4cf5bca6800523d1aafc 2023-03-24 21:55:11,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9f93ec7cd69d4cf5bca6800523d1aafc, entries=100, sequenceid=555, filesize=9.5 K 2023-03-24 21:55:11,205 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for f67314e16f9680bd83bac15d2c4afabe in 1239ms, sequenceid=555, compaction requested=true 2023-03-24 21:55:11,228 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/ead03957bc2742bfb341813291b80bc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/27572c5504da4bbbbc027a933b0a136c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/cc9794eeaeb14292b9cce778c266571b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c75d488ba7c94a7bbcdaa7a51d21a071, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d9534d3e166a4defb1423e698844dc25, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38a74fa4fa88417391a74f9705aa96bd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/e357acc6a96a4ada8f6cb4fb7a9b5739, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/5a834ac5c1e5472ca2845796e26a57d8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe] to archive 2023-03-24 21:55:11,229 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:11,230 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/7eee5321a90d473987630e91fa8358ad 2023-03-24 21:55:11,232 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/f2863931f49f43bcaa1d9fa7e345b714 2023-03-24 21:55:11,233 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/ead03957bc2742bfb341813291b80bc6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/ead03957bc2742bfb341813291b80bc6 2023-03-24 21:55:11,234 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/1c498924f7414bceb723fe6e51ab2696 2023-03-24 21:55:11,235 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df53c9134c204c3d8814e4cce4e58a57 2023-03-24 21:55:11,236 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/27572c5504da4bbbbc027a933b0a136c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/27572c5504da4bbbbc027a933b0a136c 2023-03-24 21:55:11,237 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06421227de0e45b2bf265e451c4f7488 2023-03-24 21:55:11,238 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9a6709c9c8334db2a852affefe4f3ba1 2023-03-24 21:55:11,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/15fb8571236f4f429b316d31b5f1d272 2023-03-24 21:55:11,239 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/22d72b7bd240498c86dfccf2d184cdf7 2023-03-24 21:55:11,240 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/cc9794eeaeb14292b9cce778c266571b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/cc9794eeaeb14292b9cce778c266571b 2023-03-24 21:55:11,241 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0a881d0a77c742989755ec2926d62402 2023-03-24 21:55:11,242 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/149ecce992b3400db5ad37dfbf0f756d 2023-03-24 21:55:11,243 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b9676ca9fc0f455a8282aaa2eb6397ac 2023-03-24 21:55:11,244 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c75d488ba7c94a7bbcdaa7a51d21a071 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c75d488ba7c94a7bbcdaa7a51d21a071 2023-03-24 21:55:11,245 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d0ea9a1d51a84ecdaaab099b87801869 2023-03-24 21:55:11,246 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d9534d3e166a4defb1423e698844dc25 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d9534d3e166a4defb1423e698844dc25 2023-03-24 21:55:11,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/0b8a4f7a4d8249f0b5faa1fd94438a2c 2023-03-24 21:55:11,247 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/9041d51d765349ee97f506ba04daf93c 2023-03-24 21:55:11,248 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38b74507f2cd41688a487019fbb88d3a 2023-03-24 21:55:11,249 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38a74fa4fa88417391a74f9705aa96bd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/38a74fa4fa88417391a74f9705aa96bd 2023-03-24 21:55:11,250 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/6671be2d2c5c4f7db83bce42b29b9a9d 2023-03-24 21:55:11,251 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/d04348831c2545048d7c4b47bae5671f 2023-03-24 21:55:11,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/e357acc6a96a4ada8f6cb4fb7a9b5739 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/e357acc6a96a4ada8f6cb4fb7a9b5739 2023-03-24 21:55:11,252 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/c5d1ef5939eb4950a60303dbd1b7c672 2023-03-24 21:55:11,253 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/860ab4f1a84842e0bb72cd1e7ac9f5d5 2023-03-24 21:55:11,254 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/b3be4938660b4dbaa3cbf77fc0b4cc68 2023-03-24 21:55:11,255 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/fb93b5a5a3224da994d2aeb58eac68c4 2023-03-24 21:55:11,256 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/13c4bddc0e92496898058d3af09a1cc3 2023-03-24 21:55:11,257 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/5a834ac5c1e5472ca2845796e26a57d8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/5a834ac5c1e5472ca2845796e26a57d8 2023-03-24 21:55:11,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/510c2c8e85a94359ac57a30d8c9f48de 2023-03-24 21:55:11,258 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/06659d264f6e4be3af0c8f812d0dbf58 2023-03-24 21:55:11,259 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/88a804c09ca0443eb3c7d03118a82efe 2023-03-24 21:55:11,262 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/5fe5523538824490924248a3122b0bc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/47e21716f5c34944aa3e02f03b2d15ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/c340deb0daa646529004b53a5b979a96, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d0a00886123740658fc341f420d2f57d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d7c6ba4009ec406a9332a3a3e18f375f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/bc07f0865c5a49de96bb4845c79c1f82, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e49fe5519ef34fb691d2c4512008b6c4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a03020d03710436fbd3b9d212d1f8a7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d] to archive 2023-03-24 21:55:11,263 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:11,264 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/80cc0bcf10c44df086f6a1838de31fb7 2023-03-24 21:55:11,265 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/7254cb6c82cf43db8168b613d251e9e0 2023-03-24 21:55:11,266 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/5fe5523538824490924248a3122b0bc3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/5fe5523538824490924248a3122b0bc3 2023-03-24 21:55:11,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a59daa275fab4e95800221589877622c 2023-03-24 21:55:11,267 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f3959dc8059e4ae08eec881df951349b 2023-03-24 21:55:11,268 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/47e21716f5c34944aa3e02f03b2d15ec to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/47e21716f5c34944aa3e02f03b2d15ec 2023-03-24 21:55:11,269 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dacdbe81a93e4a8282c69a0d227add59 2023-03-24 21:55:11,270 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f6d4a6e5413d4d72b17488c1a6640d9b 2023-03-24 21:55:11,271 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/26c4d9ad702b4f5dbfb43c883c292d56 2023-03-24 21:55:11,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/c340deb0daa646529004b53a5b979a96 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/c340deb0daa646529004b53a5b979a96 2023-03-24 21:55:11,272 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f295d13b7ea44669885f56869cacd79c 2023-03-24 21:55:11,273 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/93121dcb8e8d4be69a470227ac08de4d 2023-03-24 21:55:11,274 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/1ee89e500b8448b2b46a98b7d18e4700 2023-03-24 21:55:11,275 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d0a00886123740658fc341f420d2f57d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d0a00886123740658fc341f420d2f57d 2023-03-24 21:55:11,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e2953268359a42c99bf6448e58fd7b41 2023-03-24 21:55:11,276 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2c082ec855fc4cc4a3b0d03759853cb9 2023-03-24 21:55:11,277 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d7c6ba4009ec406a9332a3a3e18f375f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/d7c6ba4009ec406a9332a3a3e18f375f 2023-03-24 21:55:11,278 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4673a95902824d8da87d03ff2adc7f08 2023-03-24 21:55:11,279 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/b8b5f3d6b8df403895469e3ba1d87378 2023-03-24 21:55:11,280 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/44ca4d4d4d8b4d85adfa64a9a2463632 2023-03-24 21:55:11,281 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/bc07f0865c5a49de96bb4845c79c1f82 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/bc07f0865c5a49de96bb4845c79c1f82 2023-03-24 21:55:11,282 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/2d26cada00884ec2aa72d36372e9c023 2023-03-24 21:55:11,283 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/908956fb2b654ee8b38d94e4cefa19c1 2023-03-24 21:55:11,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e49fe5519ef34fb691d2c4512008b6c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/e49fe5519ef34fb691d2c4512008b6c4 2023-03-24 21:55:11,284 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/67467e11a31c4a15a8d50816ac9e2f54 2023-03-24 21:55:11,285 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/29060c41eedb4538a29c25e15b6ba871 2023-03-24 21:55:11,286 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/dfa959328a854b588b4bd27339d1f814 2023-03-24 21:55:11,287 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/da53f5e207b840b3bdbf018946025d2c 2023-03-24 21:55:11,288 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0669676b7a4e4e60a3a2b0003acc3e02 2023-03-24 21:55:11,289 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a03020d03710436fbd3b9d212d1f8a7b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/a03020d03710436fbd3b9d212d1f8a7b 2023-03-24 21:55:11,290 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/ccd5a15227f14b5ba512c57444b97186 2023-03-24 21:55:11,291 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/3fff4b6dbe6e4c4fbbafa4bdf980fcdf 2023-03-24 21:55:11,292 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/0d8375ff02d44f1f991bbfe4ea77e60d 2023-03-24 21:55:11,295 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/47bac44890f94352b8334786d5a8a583, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/17de76a7534a4414bccfdc71146e2fd9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/53a12f1661b44c348ca4da84bcd4f0f2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6770d72d283d4804a4e272e15f5eee86, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/bed5684ab5aa4bbe90c342765b74bc5e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/45bffbfc50884f25b2cc41375c3bbf30, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7602e2021366477aa9d5ee97605fc69e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6466fa6a8d0c43fd8f4188c47bd4ae40, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01] to archive 2023-03-24 21:55:11,296 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:11,297 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/83e8a94ae11941e0ad13bc8458d24bf8 2023-03-24 21:55:11,298 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/73f12b58f1b448ab8663d5ec24894bb1 2023-03-24 21:55:11,299 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/47bac44890f94352b8334786d5a8a583 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/47bac44890f94352b8334786d5a8a583 2023-03-24 21:55:11,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/2bc173268b774a199d7d5ddb221399d3 2023-03-24 21:55:11,300 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ef20ce46513342eeb6937f6bcd514574 2023-03-24 21:55:11,301 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/17de76a7534a4414bccfdc71146e2fd9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/17de76a7534a4414bccfdc71146e2fd9 2023-03-24 21:55:11,302 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8dbcefe8f80840919ac054202812f27d 2023-03-24 21:55:11,303 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e3de1a3edf824bd9bb8db3b95e4c9868 2023-03-24 21:55:11,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d744729fe57a48b991fbf3bee6450ab6 2023-03-24 21:55:11,304 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/53a12f1661b44c348ca4da84bcd4f0f2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/53a12f1661b44c348ca4da84bcd4f0f2 2023-03-24 21:55:11,305 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/720e253c0f7e489a9909243161f7bcf9 2023-03-24 21:55:11,306 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7879c4957d844f859cb204b6309f2e87 2023-03-24 21:55:11,307 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/ae5861444de24de2a641cfb240bd1d74 2023-03-24 21:55:11,308 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6770d72d283d4804a4e272e15f5eee86 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6770d72d283d4804a4e272e15f5eee86 2023-03-24 21:55:11,309 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/352e131447b442a4a3c61f4a20fe812f 2023-03-24 21:55:11,310 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/b8d88e125b3745fcb4e5ba63efc55769 2023-03-24 21:55:11,311 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/bed5684ab5aa4bbe90c342765b74bc5e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/bed5684ab5aa4bbe90c342765b74bc5e 2023-03-24 21:55:11,312 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/dd1f26e02aec40e896200f227688f457 2023-03-24 21:55:11,313 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/e4e4ff72183d4137b9c02e7a769b9095 2023-03-24 21:55:11,314 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/a1c5ce1171a940fbb8976b32cefd8562 2023-03-24 21:55:11,315 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/45bffbfc50884f25b2cc41375c3bbf30 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/45bffbfc50884f25b2cc41375c3bbf30 2023-03-24 21:55:11,316 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9517fb8e3c194d53820a09e2bc4b891e 2023-03-24 21:55:11,316 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/686da21ce349457ca950ef376d5795ce 2023-03-24 21:55:11,317 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7602e2021366477aa9d5ee97605fc69e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7602e2021366477aa9d5ee97605fc69e 2023-03-24 21:55:11,318 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/670d520e2be6460b8562214e2f5a6cf7 2023-03-24 21:55:11,319 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/4d46b165fea64f9b852580ee1a706710 2023-03-24 21:55:11,320 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/62ad9f5123ad4aef93a05d49d8bbea2e 2023-03-24 21:55:11,321 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/7b5129a56bd0478e9e29ade19d144efa 2023-03-24 21:55:11,322 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6a4a930d68cc4754bd03faea747e53e8 2023-03-24 21:55:11,323 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6466fa6a8d0c43fd8f4188c47bd4ae40 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/6466fa6a8d0c43fd8f4188c47bd4ae40 2023-03-24 21:55:11,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9a6778ce97bc4870996d40891a668d7b 2023-03-24 21:55:11,324 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/d3917adb5be94e1ea352eb86c7f53e80 2023-03-24 21:55:11,325 DEBUG [StoreCloser-TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8cda9242e36942d69f89b8a1b846ab01 2023-03-24 21:55:11,329 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/recovered.edits/558.seqid, newMaxSeqId=558, maxSeqId=1 2023-03-24 21:55:11,330 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe. 2023-03-24 21:55:11,330 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for f67314e16f9680bd83bac15d2c4afabe: 2023-03-24 21:55:11,332 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:11,332 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=f67314e16f9680bd83bac15d2c4afabe, regionState=CLOSED 2023-03-24 21:55:11,332 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694911332"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694911332"}]},"ts":"1679694911332"} 2023-03-24 21:55:11,334 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-24 21:55:11,334 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure f67314e16f9680bd83bac15d2c4afabe, server=jenkins-hbase9.apache.org,45673,1679694819052 in 1.5190 sec 2023-03-24 21:55:11,335 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-24 21:55:11,335 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=f67314e16f9680bd83bac15d2c4afabe, UNASSIGN in 1.5230 sec 2023-03-24 21:55:11,336 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694911335"}]},"ts":"1679694911335"} 2023-03-24 21:55:11,336 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:55:11,338 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:55:11,339 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5340 sec 2023-03-24 21:55:11,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-24 21:55:11,911 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-24 21:55:11,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:55:11,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,915 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,916 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 21:55:11,920 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:11,922 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/recovered.edits] 2023-03-24 21:55:11,924 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/24f2b8ad5d204ea08d73748b6e7eb58c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/24f2b8ad5d204ea08d73748b6e7eb58c 2023-03-24 21:55:11,926 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/331bddb069c84b25b4277d7b7c69e382 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/331bddb069c84b25b4277d7b7c69e382 2023-03-24 21:55:11,927 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df9a45042ad643468b751c4785e9b00f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/A/df9a45042ad643468b751c4785e9b00f 2023-03-24 21:55:11,929 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4877d23f7ff54fe6b2bfb910fddfe1f7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/4877d23f7ff54fe6b2bfb910fddfe1f7 2023-03-24 21:55:11,930 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/8139121778ad48f08048e730cafa182e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/8139121778ad48f08048e730cafa182e 2023-03-24 21:55:11,931 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f08a952e70b9450e97089e1465c49cb9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/B/f08a952e70b9450e97089e1465c49cb9 2023-03-24 21:55:11,933 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/186bc5c99455404db9af63a6b03d85f7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/186bc5c99455404db9af63a6b03d85f7 2023-03-24 21:55:11,934 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8f7e6206877d462ea483fcfa169a9d66 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/8f7e6206877d462ea483fcfa169a9d66 2023-03-24 21:55:11,935 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9f93ec7cd69d4cf5bca6800523d1aafc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/C/9f93ec7cd69d4cf5bca6800523d1aafc 2023-03-24 21:55:11,938 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/recovered.edits/558.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe/recovered.edits/558.seqid 2023-03-24 21:55:11,939 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/f67314e16f9680bd83bac15d2c4afabe 2023-03-24 21:55:11,939 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:55:11,941 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,942 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:55:11,943 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:55:11,944 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,944 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:55:11,944 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694911944"}]},"ts":"9223372036854775807"} 2023-03-24 21:55:11,946 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:55:11,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => f67314e16f9680bd83bac15d2c4afabe, NAME => 'TestAcidGuarantees,,1679694883542.f67314e16f9680bd83bac15d2c4afabe.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:55:11,946 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:55:11,946 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694911946"}]},"ts":"9223372036854775807"} 2023-03-24 21:55:11,947 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:55:11,948 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:11,949 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-03-24 21:55:12,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-24 21:55:12,017 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-24 21:55:12,028 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=226 (was 224) - Thread LEAK? -, OpenFileDescriptor=531 (was 525) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=318 (was 358), ProcessCount=169 (was 169), AvailableMemoryMB=7931 (was 8128) 2023-03-24 21:55:12,040 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=226, OpenFileDescriptor=531, MaxFileDescriptor=60000, SystemLoadAverage=318, ProcessCount=169, AvailableMemoryMB=7930 2023-03-24 21:55:12,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:55:12,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:55:12,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:12,045 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:55:12,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-24 21:55:12,046 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:55:12,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:12,048 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:12,048 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 empty. 2023-03-24 21:55:12,049 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:12,049 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:55:12,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:12,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:12,456 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:55:12,457 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 3b6e2d0df6ef7b528783b27726388c56, NAME => 'TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:55:12,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 3b6e2d0df6ef7b528783b27726388c56, disabling compactions & flushes 2023-03-24 21:55:12,863 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. after waiting 0 ms 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:12,863 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:12,863 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:12,865 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:55:12,866 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694912866"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694912866"}]},"ts":"1679694912866"} 2023-03-24 21:55:12,867 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:55:12,868 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:55:12,868 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694912868"}]},"ts":"1679694912868"} 2023-03-24 21:55:12,868 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:55:12,871 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, ASSIGN}] 2023-03-24 21:55:12,873 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, ASSIGN 2023-03-24 21:55:12,873 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:55:13,024 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:13,024 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694913024"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694913024"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694913024"}]},"ts":"1679694913024"} 2023-03-24 21:55:13,026 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:13,180 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:13,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3b6e2d0df6ef7b528783b27726388c56, NAME => 'TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:55:13,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:55:13,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,182 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,183 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A 2023-03-24 21:55:13,183 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A 2023-03-24 21:55:13,183 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:13,184 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName A 2023-03-24 21:55:13,184 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:13,184 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,185 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B 2023-03-24 21:55:13,185 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B 2023-03-24 21:55:13,186 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:13,186 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName B 2023-03-24 21:55:13,186 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:13,186 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,187 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C 2023-03-24 21:55:13,187 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C 2023-03-24 21:55:13,187 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:13,188 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName C 2023-03-24 21:55:13,188 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:13,188 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:13,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,189 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,190 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:55:13,191 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:13,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:55:13,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 3b6e2d0df6ef7b528783b27726388c56; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69618132, jitterRate=0.03739100694656372}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:55:13,193 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:13,193 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., pid=37, masterSystemTime=1679694913177 2023-03-24 21:55:13,195 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:13,195 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:13,195 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:13,195 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694913195"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694913195"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694913195"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694913195"}]},"ts":"1679694913195"} 2023-03-24 21:55:13,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-24 21:55:13,197 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 in 170 msec 2023-03-24 21:55:13,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-24 21:55:13,199 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, ASSIGN in 326 msec 2023-03-24 21:55:13,199 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:55:13,199 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694913199"}]},"ts":"1679694913199"} 2023-03-24 21:55:13,200 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:55:13,202 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:55:13,203 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1590 sec 2023-03-24 21:55:14,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-24 21:55:14,151 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-24 21:55:14,151 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x016ba88c to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:14,156 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@117dbb9a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:14,157 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:14,159 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36930, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:14,160 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:55:14,161 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37246, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:55:14,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:55:14,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$12(2454): Client=jenkins//172.31.2.10 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:55:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:14,572 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:55:14,573 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:55:14,573 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:55:14,575 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 21:55:14,577 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, REOPEN/MOVE}] 2023-03-24 21:55:14,578 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, REOPEN/MOVE 2023-03-24 21:55:14,578 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:14,578 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694914578"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694914578"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694914578"}]},"ts":"1679694914578"} 2023-03-24 21:55:14,579 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:14,731 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:14,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 3b6e2d0df6ef7b528783b27726388c56, disabling compactions & flushes 2023-03-24 21:55:14,732 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:14,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:14,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. after waiting 0 ms 2023-03-24 21:55:14,732 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:14,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 21:55:14,735 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:14,735 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:14,735 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 3b6e2d0df6ef7b528783b27726388c56 to self. 2023-03-24 21:55:14,737 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:14,737 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=CLOSED 2023-03-24 21:55:14,737 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694914737"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694914737"}]},"ts":"1679694914737"} 2023-03-24 21:55:14,739 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-24 21:55:14,739 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 in 159 msec 2023-03-24 21:55:14,740 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=true 2023-03-24 21:55:14,890 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:14,890 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694914890"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694914890"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694914890"}]},"ts":"1679694914890"} 2023-03-24 21:55:14,891 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:15,046 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:15,046 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 3b6e2d0df6ef7b528783b27726388c56, NAME => 'TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:55:15,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:55:15,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,047 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,048 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,049 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A 2023-03-24 21:55:15,049 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A 2023-03-24 21:55:15,050 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:15,050 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName A 2023-03-24 21:55:15,051 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:15,051 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,052 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B 2023-03-24 21:55:15,052 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B 2023-03-24 21:55:15,052 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:15,052 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName B 2023-03-24 21:55:15,053 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:15,053 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,054 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C 2023-03-24 21:55:15,054 DEBUG [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C 2023-03-24 21:55:15,054 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:15,054 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 3b6e2d0df6ef7b528783b27726388c56 columnFamilyName C 2023-03-24 21:55:15,054 INFO [StoreOpener-3b6e2d0df6ef7b528783b27726388c56-1] regionserver.HStore(310): Store=3b6e2d0df6ef7b528783b27726388c56/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:15,054 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:15,055 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,056 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,058 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:55:15,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,059 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 3b6e2d0df6ef7b528783b27726388c56; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62067325, jitterRate=-0.07512478530406952}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:55:15,059 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:15,060 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., pid=42, masterSystemTime=1679694915043 2023-03-24 21:55:15,061 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:15,061 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:15,061 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,062 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694915061"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694915061"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694915061"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694915061"}]},"ts":"1679694915061"} 2023-03-24 21:55:15,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-24 21:55:15,063 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 in 171 msec 2023-03-24 21:55:15,064 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-24 21:55:15,064 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, REOPEN/MOVE in 486 msec 2023-03-24 21:55:15,066 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-24 21:55:15,066 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-03-24 21:55:15,067 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-03-24 21:55:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-24 21:55:15,069 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x2879fb80 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,076 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748a19cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,077 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x3ded5508 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,081 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5aaf0cce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,082 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x484332e6 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,086 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f517bf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,087 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x3be39f87 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,092 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6ae64e1f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,092 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x154073b2 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,098 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3278c93a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,099 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x636cfa8a to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,103 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@17a9cec4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,104 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x01035478 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,110 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53f5678, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,110 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x72f4141b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,118 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69cf3ea9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,118 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x01ad7293 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,132 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@53abea71, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,133 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x34018a0c to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:15,137 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e525ff5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:15,140 DEBUG [hconnection-0x536a4c39-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,142 DEBUG [hconnection-0x6e6b86cc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,144 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36940, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:15,151 DEBUG [hconnection-0x1db04254-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,156 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36954, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,157 DEBUG [hconnection-0x310cc13f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,157 DEBUG [hconnection-0x6f1c59bb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,157 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36962, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,157 DEBUG [hconnection-0x29015de1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,158 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36976, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,159 DEBUG [hconnection-0x6c7083d6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,160 DEBUG [hconnection-0x75aadbae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,160 DEBUG [hconnection-0x2090ce0d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,160 DEBUG [hconnection-0x5cdd7655-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:15,161 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:36986, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:15,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:15,167 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:15,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:15,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:15,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:15,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:15,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:15,169 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37004, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,169 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37026, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,170 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:15,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:15,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:15,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:15,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:15,171 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:15,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:15,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:15,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:15,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:15,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:15,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:15,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:15,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:15,175 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:15,175 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:15,175 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:15,178 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 21:55:15,179 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37012, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,180 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37000, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,180 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37018, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:15,187 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:15,187 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:15,188 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:15,188 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:15,188 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:15,188 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:15,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:15,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:15,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694915194, max timestamp=1679694915207Num uniques 859; 2023-03-24 21:55:15,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:15,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694915194, max timestamp=1679694915207Num uniques 859; 2023-03-24 21:55:15,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:15,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694915194, max timestamp=1679694915207Num uniques 859; 2023-03-24 21:55:15,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:15,210 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f9f97f4fc9f14fbfbb6e3e200034de44 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f9f97f4fc9f14fbfbb6e3e200034de44 2023-03-24 21:55:15,215 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=12, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/2e6427ecfbe84cffa90f306b94d21301 2023-03-24 21:55:15,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694975223, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694975228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12263, totalHeapSize=28112, min timestamp=1679694915219, max timestamp=1679694915229Num uniques 1068; 2023-03-24 21:55:15,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=48903, new segment data size=12263, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 21:55:15,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694915219, max timestamp=1679694915229Num uniques 1068; 2023-03-24 21:55:15,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12262, totalHeapSize=28112, min timestamp=1679694915219, max timestamp=1679694915229Num uniques 1068; 2023-03-24 21:55:15,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 21:55:15,232 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48902, new segment data size=12262, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-24 21:55:15,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694975246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,250 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694975249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694975249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694975331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694975332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694975350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694975351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,356 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694975351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694975539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694975539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694975557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694975558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,562 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694975558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,637 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/8ac7eaf333af4fdb8172aaf6c457446a 2023-03-24 21:55:15,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694975844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694975844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694975863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694975863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:15,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:15,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694975864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,049 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/8bb634433c1b423e90c9906eaf40c810 2023-03-24 21:55:16,054 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/2e6427ecfbe84cffa90f306b94d21301 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301 2023-03-24 21:55:16,058 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301, entries=150, sequenceid=12, filesize=25.1 K 2023-03-24 21:55:16,059 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/8ac7eaf333af4fdb8172aaf6c457446a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a 2023-03-24 21:55:16,063 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 21:55:16,063 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/8bb634433c1b423e90c9906eaf40c810 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810 2023-03-24 21:55:16,067 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810, entries=150, sequenceid=12, filesize=11.6 K 2023-03-24 21:55:16,067 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 3b6e2d0df6ef7b528783b27726388c56 in 889ms, sequenceid=12, compaction requested=false 2023-03-24 21:55:16,067 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-24 21:55:16,069 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:16,070 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:16,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:16,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:16,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:16,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:16,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,071 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:16,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:16,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:16,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:16,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a28547e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:16,073 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:16,073 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,074 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:16,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:16,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:16,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:16,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,076 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:16,076 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:16,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:16,076 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:16,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:16,079 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:16,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:16,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:16,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:16,079 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:16,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:16,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:16,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:16,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:16,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:16,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:16,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:16,087 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:16,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:16,087 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:16,087 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:16,087 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:16,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:16,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:16,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:16,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:16,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:16,187 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:16,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:16,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:16,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:16,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:16,190 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:16,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:16,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:16,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:16,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:16,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,197 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:16,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:16,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:16,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:16,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:16,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:16,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:16,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:16,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:16,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:16,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:16,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:16,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:16,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:16,199 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:16,199 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:16,199 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=147.60 KB heapSize=414.61 KB 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:16,200 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:16,353 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:16,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694976356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,368 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694976358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694976367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694976369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,380 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694976373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694976459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694976469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,620 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249e5ff70f4a93483aa16071d0c5e4ccd7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e5ff70f4a93483aa16071d0c5e4ccd7 2023-03-24 21:55:16,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694976664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694976671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,865 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:55:16,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694976970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:16,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:16,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694976976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,025 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=66, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/ebdc2074d42441a79d5d018fb6aac675 2023-03-24 21:55:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694977379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694977379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694977385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,435 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/9af7332c75ca4b8384fe8e3ff9bb447a 2023-03-24 21:55:17,463 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/b9b1b61d5ae94d5ea5235e8b7a643b1d 2023-03-24 21:55:17,468 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/ebdc2074d42441a79d5d018fb6aac675 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675 2023-03-24 21:55:17,472 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675, entries=150, sequenceid=66, filesize=25.1 K 2023-03-24 21:55:17,473 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/9af7332c75ca4b8384fe8e3ff9bb447a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a 2023-03-24 21:55:17,476 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a, entries=150, sequenceid=66, filesize=11.6 K 2023-03-24 21:55:17,477 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/b9b1b61d5ae94d5ea5235e8b7a643b1d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d 2023-03-24 21:55:17,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694977475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,480 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d, entries=150, sequenceid=66, filesize=11.6 K 2023-03-24 21:55:17,481 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.56 KB/424512, currentSize=33.54 KB/34350 for 3b6e2d0df6ef7b528783b27726388c56 in 1282ms, sequenceid=66, compaction requested=false 2023-03-24 21:55:17,481 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:17,481 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:17,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:17,483 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:17,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a5492c5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:17,485 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:17,485 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,488 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:17,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:17,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:17,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:17,491 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:17,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:17,493 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:17,493 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:17,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:17,497 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:17,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:17,497 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:17,500 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,500 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,500 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,501 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,508 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:17,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:17,508 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,508 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,508 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:17,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,508 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:17,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:17,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,608 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:17,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:17,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694917564, max timestamp=1679694917608Num uniques 859; 2023-03-24 21:55:17,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:17,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694917564, max timestamp=1679694917608Num uniques 859; 2023-03-24 21:55:17,610 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:17,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:17,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694917564, max timestamp=1679694917608Num uniques 859; 2023-03-24 21:55:17,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:17,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:17,614 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:17,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:17,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:17,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:17,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:17,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:17,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,621 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:17,625 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,625 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:17,625 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:17,625 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:17,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:17,627 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,627 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:17,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c74a93c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:17,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:17,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,631 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,632 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:17,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:17,632 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:17,632 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:17,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:17,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:17,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:17,637 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:17,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:17,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:17,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:17,637 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:17,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,640 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,641 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:17,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,646 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:17,646 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,646 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,746 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:17,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:17,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694977741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:17,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:17,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:17,750 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:17,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:17,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:17,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:17,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:17,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:17,753 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:17,753 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:17,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:17,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:17,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:17,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:17,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:17,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:17,757 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,757 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:17,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:17,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d51c1c5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:17,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:17,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,762 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:17,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:17,762 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:17,762 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:17,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:17,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:17,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:17,766 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:17,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:17,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:17,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:17,767 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:17,770 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,770 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,770 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:17,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:17,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,779 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:17,779 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:17,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,856 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:17,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694977849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,879 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:17,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:17,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:17,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:17,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:17,884 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:17,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:17,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:17,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,885 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,886 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:17,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:17,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:17,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:17,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:17,887 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:17,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:17,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:17,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45c2e5b5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:17,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:17,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,898 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:17,898 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:17,898 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:17,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,898 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:17,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:17,900 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,900 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:17,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:17,902 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:17,902 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:17,902 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:17,902 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:17,902 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:17,903 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:17,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:17,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,905 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,905 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,907 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:17,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:17,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:17,910 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,910 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:17,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:17,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:17,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:17,939 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c8ec1dd526754fb183c4f7f18645a852 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c8ec1dd526754fb183c4f7f18645a852 2023-03-24 21:55:18,011 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,014 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,015 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,015 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,018 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,018 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,018 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,018 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,023 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bae9572[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,032 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,032 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,038 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,038 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:18,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694978059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,139 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,142 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,143 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,146 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,146 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,146 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,146 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@621afc72[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,156 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,156 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,165 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,165 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,265 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,268 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,273 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,273 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,273 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,273 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,273 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,274 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3196461d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,279 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,286 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,286 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,292 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,292 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,345 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=76, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/66a2c5d65de34beeb2706bb863a5393b 2023-03-24 21:55:18,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:18,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694978366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,392 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,395 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,397 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,397 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,398 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,398 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,399 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,400 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,401 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,401 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@632bf2d7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,409 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,409 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,411 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,415 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,415 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,415 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,489 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:18,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694978486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,515 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,518 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,519 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,521 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,521 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,521 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,521 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,522 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,524 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,525 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,525 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cb1eec5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,529 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,530 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,534 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,534 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,534 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,536 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,536 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,536 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,536 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,541 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,541 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,541 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,641 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,647 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,647 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,649 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,649 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,650 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,650 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,651 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,651 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,652 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,653 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,653 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49fd744d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,656 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,658 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,658 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,661 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,661 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,661 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,661 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,661 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,666 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,666 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,667 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,667 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,669 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,671 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,672 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,672 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,672 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,672 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,757 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/8e2bd0ec32e94b5da72b5e57e70b6ea2 2023-03-24 21:55:18,772 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,775 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,777 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,779 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,781 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,781 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5865c1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,784 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,784 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,785 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,788 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,788 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,788 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,792 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,795 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,795 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,796 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,796 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:18,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694978872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,896 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:18,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:18,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:18,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:18,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:18,899 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:18,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:18,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:18,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:18,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:18,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:18,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:18,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:18,902 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:18,902 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:18,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:18,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76598b17[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:18,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:18,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:18,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:18,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:18,908 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:18,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:18,910 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:18,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:18,912 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:18,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:18,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:18,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:18,912 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:18,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:18,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:18,914 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,914 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:18,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,915 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:18,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:18,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:18,918 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:18,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:18,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:18,918 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:18,918 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,019 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,022 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,022 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,024 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,026 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,026 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,027 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,028 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,029 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,030 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d84df[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,031 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,033 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,035 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,035 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,044 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,044 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,144 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,148 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,149 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,149 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,151 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,151 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,151 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,151 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2710d723[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,156 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,161 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,161 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=76 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/2f53ffbc665c4c2bbb1abd5b45d96043 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,168 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,168 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,168 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:19,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,174 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/66a2c5d65de34beeb2706bb863a5393b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b 2023-03-24 21:55:19,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b, entries=200, sequenceid=76, filesize=31.9 K 2023-03-24 21:55:19,178 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/8e2bd0ec32e94b5da72b5e57e70b6ea2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2 2023-03-24 21:55:19,183 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2, entries=150, sequenceid=76, filesize=11.6 K 2023-03-24 21:55:19,183 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/2f53ffbc665c4c2bbb1abd5b45d96043 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043 2023-03-24 21:55:19,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043, entries=150, sequenceid=76, filesize=11.6 K 2023-03-24 21:55:19,205 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 3b6e2d0df6ef7b528783b27726388c56 in 1712ms, sequenceid=76, compaction requested=true 2023-03-24 21:55:19,205 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,206 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:19,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:19,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:19,206 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:19,206 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:19,209 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:19,209 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:19,209 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,210 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=82.2 K 2023-03-24 21:55:19,210 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:19,210 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2e6427ecfbe84cffa90f306b94d21301, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679694915149 2023-03-24 21:55:19,210 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:19,210 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,210 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=34.8 K 2023-03-24 21:55:19,211 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ebdc2074d42441a79d5d018fb6aac675, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679694915245 2023-03-24 21:55:19,211 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8ac7eaf333af4fdb8172aaf6c457446a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679694915149 2023-03-24 21:55:19,211 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9af7332c75ca4b8384fe8e3ff9bb447a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679694915245 2023-03-24 21:55:19,211 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 66a2c5d65de34beeb2706bb863a5393b, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679694916352 2023-03-24 21:55:19,212 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8e2bd0ec32e94b5da72b5e57e70b6ea2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679694916352 2023-03-24 21:55:19,236 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#316 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:19,256 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#315 average throughput is 0.68 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:19,268 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,279 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,281 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,282 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,295 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,295 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,295 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:55:19,295 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:19,295 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:19,295 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:19,296 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:19,296 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:19,296 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:19,336 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240b9333fba47c4dc39ce18d0b31d0ba5b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240b9333fba47c4dc39ce18d0b31d0ba5b 2023-03-24 21:55:19,351 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=113, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/cc67e8c75f234b54856888737e933234 2023-03-24 21:55:19,371 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/60ff7b3cba9f48e6bbe6312c47b7b002 2023-03-24 21:55:19,395 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/ff1b6dc6332e4cd88d970e1f52466053 2023-03-24 21:55:19,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:19,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,399 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/cc67e8c75f234b54856888737e933234 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234 2023-03-24 21:55:19,402 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234, entries=150, sequenceid=113, filesize=25.1 K 2023-03-24 21:55:19,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/60ff7b3cba9f48e6bbe6312c47b7b002 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002 2023-03-24 21:55:19,409 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002, entries=150, sequenceid=113, filesize=11.6 K 2023-03-24 21:55:19,409 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/ff1b6dc6332e4cd88d970e1f52466053 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053 2023-03-24 21:55:19,416 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053, entries=150, sequenceid=113, filesize=11.6 K 2023-03-24 21:55:19,417 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=40.25 KB/41220 for 3b6e2d0df6ef7b528783b27726388c56 in 122ms, sequenceid=113, compaction requested=true 2023-03-24 21:55:19,417 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,417 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,424 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e6fd86d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:19,441 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:19,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:19,444 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,446 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,447 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,460 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,460 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,493 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f3e5a013ddda44f0941541922a5c6420 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f3e5a013ddda44f0941541922a5c6420 2023-03-24 21:55:19,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694979498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694979499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,512 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694979506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,560 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,567 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,567 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,567 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,567 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d4ff84[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,583 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,583 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,594 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,594 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:19,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694979607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694979607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694979613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,673 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/8af3d20d45cb495281f4a77e4f1a90a5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8af3d20d45cb495281f4a77e4f1a90a5 2023-03-24 21:55:19,677 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into 8af3d20d45cb495281f4a77e4f1a90a5(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:19,678 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,678 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=13, startTime=1679694919206; duration=0sec 2023-03-24 21:55:19,678 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:19,678 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:19,679 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:19,679 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/C is initiating minor compaction (all files) 2023-03-24 21:55:19,679 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/C in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,679 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=46.4 K 2023-03-24 21:55:19,679 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8bb634433c1b423e90c9906eaf40c810, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1679694915149 2023-03-24 21:55:19,680 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b9b1b61d5ae94d5ea5235e8b7a643b1d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679694915245 2023-03-24 21:55:19,680 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2f53ffbc665c4c2bbb1abd5b45d96043, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679694916352 2023-03-24 21:55:19,680 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ff1b6dc6332e4cd88d970e1f52466053, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694917690 2023-03-24 21:55:19,689 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#C#compaction#321 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:19,694 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,705 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,709 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/b0df0325236e4052aa77547baa73b66e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b0df0325236e4052aa77547baa73b66e 2023-03-24 21:55:19,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,710 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,711 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,711 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,711 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,711 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,714 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into b0df0325236e4052aa77547baa73b66e(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:19,714 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,714 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=13, startTime=1679694919206; duration=0sec 2023-03-24 21:55:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,714 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d76cabd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,718 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,720 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,720 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,728 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,728 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,728 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694979810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694979811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694979818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,828 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,832 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,833 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:19,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:19,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:19,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,837 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:19,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfeb3ff[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:19,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:19,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,841 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:19,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:19,841 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:19,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:19,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:19,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:19,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:19,846 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:19,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:19,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:19,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:19,846 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:19,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:19,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:19,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:19,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:19,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:19,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:19,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:19,852 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:19,852 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:19,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:19,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:19,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694979884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,903 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=125, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/c10510f4d88e4dc09caaa96a8b022505 2023-03-24 21:55:19,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/1a64e628d7744535a4f1f889d7bc6021 2023-03-24 21:55:19,921 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=125 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/fefab3a6b1cc4af9a47a10bc336b025f 2023-03-24 21:55:19,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/c10510f4d88e4dc09caaa96a8b022505 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505 2023-03-24 21:55:19,927 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505, entries=450, sequenceid=125, filesize=65.8 K 2023-03-24 21:55:19,928 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/1a64e628d7744535a4f1f889d7bc6021 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021 2023-03-24 21:55:19,932 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021, entries=150, sequenceid=125, filesize=11.6 K 2023-03-24 21:55:19,933 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/fefab3a6b1cc4af9a47a10bc336b025f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f 2023-03-24 21:55:19,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f, entries=150, sequenceid=125, filesize=11.6 K 2023-03-24 21:55:19,937 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3b6e2d0df6ef7b528783b27726388c56 in 496ms, sequenceid=125, compaction requested=true 2023-03-24 21:55:19,937 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:19,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:19,937 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:19,937 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:55:19,938 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 118989 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:19,938 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:19,938 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,938 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b0df0325236e4052aa77547baa73b66e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=116.2 K 2023-03-24 21:55:19,939 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b0df0325236e4052aa77547baa73b66e, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679694916352 2023-03-24 21:55:19,939 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cc67e8c75f234b54856888737e933234, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694917690 2023-03-24 21:55:19,939 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c10510f4d88e4dc09caaa96a8b022505, keycount=450, bloomtype=ROW, size=65.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679694919394 2023-03-24 21:55:19,951 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#324 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:19,952 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:19,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:19,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:19,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:19,958 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:19,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:19,958 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:19,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,962 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:19,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:19,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:19,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:19,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:19,964 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:19,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:19,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:19,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:19,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:19,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:19,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:19,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:19,967 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,967 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:19,968 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:19,968 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:19,984 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/8945ec32c21d4ace80c087ad68568d78 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/8945ec32c21d4ace80c087ad68568d78 2023-03-24 21:55:19,988 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into 8945ec32c21d4ace80c087ad68568d78(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:19,988 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:19,988 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=13, startTime=1679694919937; duration=0sec 2023-03-24 21:55:19,988 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:55:19,988 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-03-24 21:55:19,989 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 21:55:19,989 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 21:55:19,989 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. because compaction request was cancelled 2023-03-24 21:55:19,989 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:19,990 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:19,990 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:19,990 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:19,990 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8af3d20d45cb495281f4a77e4f1a90a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=34.9 K 2023-03-24 21:55:19,990 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8af3d20d45cb495281f4a77e4f1a90a5, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=76, earliestPutTs=1679694916352 2023-03-24 21:55:19,991 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 60ff7b3cba9f48e6bbe6312c47b7b002, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694917690 2023-03-24 21:55:19,991 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1a64e628d7744535a4f1f889d7bc6021, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679694919405 2023-03-24 21:55:19,997 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#326 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:20,105 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/381fc0a1851040619756e1c1edd76cd0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/381fc0a1851040619756e1c1edd76cd0 2023-03-24 21:55:20,109 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/C of 3b6e2d0df6ef7b528783b27726388c56 into 381fc0a1851040619756e1c1edd76cd0(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:20,109 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:20,109 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/C, priority=12, startTime=1679694919206; duration=0sec 2023-03-24 21:55:20,109 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:20,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:20,119 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:20,132 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694980129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694980130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694980130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694980233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694980236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694980236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,385 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b43581455a3d423183f59f4dd2a09fe0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b43581455a3d423183f59f4dd2a09fe0 2023-03-24 21:55:20,392 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/e9f4b53dbb914c06a7099f168539fd4e 2023-03-24 21:55:20,415 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/3a0d88b4557e450bab8b12c4d76075a1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/3a0d88b4557e450bab8b12c4d76075a1 2023-03-24 21:55:20,420 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into 3a0d88b4557e450bab8b12c4d76075a1(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:20,420 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:20,420 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=13, startTime=1679694919937; duration=0sec 2023-03-24 21:55:20,421 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:20,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694980438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694980438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694980442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694980509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,513 DEBUG [Thread-2372] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:20,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694980743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,745 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694980743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:20,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694980751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:20,802 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/5f76b6c5982a451d880a52c8d52de902 2023-03-24 21:55:21,212 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/6cf6b2593e564195ba8536731fd8ce6e 2023-03-24 21:55:21,216 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/e9f4b53dbb914c06a7099f168539fd4e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e 2023-03-24 21:55:21,219 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e, entries=150, sequenceid=147, filesize=25.3 K 2023-03-24 21:55:21,220 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/5f76b6c5982a451d880a52c8d52de902 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902 2023-03-24 21:55:21,224 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902, entries=150, sequenceid=147, filesize=11.8 K 2023-03-24 21:55:21,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/6cf6b2593e564195ba8536731fd8ce6e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e 2023-03-24 21:55:21,228 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e, entries=150, sequenceid=147, filesize=11.8 K 2023-03-24 21:55:21,228 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3b6e2d0df6ef7b528783b27726388c56 in 1261ms, sequenceid=147, compaction requested=true 2023-03-24 21:55:21,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:21,228 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,229 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,229 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,230 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,230 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,230 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c82b4bd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,235 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,235 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,235 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,235 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,237 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,237 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,240 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,240 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,240 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,243 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,244 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,244 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,244 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:21,252 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:21,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:21,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694981282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,291 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694981283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694981288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,344 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,348 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,349 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,350 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,351 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,351 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,351 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,351 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,351 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,351 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,352 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,352 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,352 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,352 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,352 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,352 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,352 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,352 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f194276[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,361 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,363 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,366 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,366 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,371 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,371 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,371 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,371 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,372 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,372 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,393 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694981389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694981392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,398 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694981394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,472 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,475 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,476 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,476 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,476 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,476 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,477 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,478 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,478 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,479 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,480 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,480 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,480 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,480 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,480 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,481 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7471540a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,486 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,486 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,487 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,489 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,489 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,491 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,491 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,493 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,495 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,497 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,498 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,498 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,498 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,598 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694981595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,601 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,601 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,602 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694981597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,603 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,604 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,605 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,605 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,605 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,605 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,605 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694981599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,611 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d396030[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,612 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,615 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,615 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,615 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,616 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,618 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,618 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,619 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,620 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,625 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,625 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,626 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,664 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249f1d9821809e4653a010fb01c7eb4ac5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f1d9821809e4653a010fb01c7eb4ac5 2023-03-24 21:55:21,669 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=163, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/3b70ee8f1d1745e2aa1c168355f39a8e 2023-03-24 21:55:21,726 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,729 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,730 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,733 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,735 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44070076[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,738 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,738 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,738 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,743 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,743 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,745 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,745 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,748 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,749 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,752 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,752 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,852 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,855 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,859 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,859 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,860 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,860 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,862 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3178e072[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,864 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,864 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,865 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,867 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,867 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,867 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,869 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,869 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,876 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,877 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,877 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:21,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:21,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694981900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694981900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,906 DEBUG [Thread-2374] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4163 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:21,907 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694981906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:21,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694981907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,977 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:21,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:21,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:21,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:21,980 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,981 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:21,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:21,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:21,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:21,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:21,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:21,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:21,983 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:21,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:21,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:21,983 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:21,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:21,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:21,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:21,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:21,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:21,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fd72353[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:21,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:21,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,988 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:21,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:21,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:21,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:21,988 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:21,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:21,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:21,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,991 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:21,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:21,994 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:21,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:21,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:21,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:21,994 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:21,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:21,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:21,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:21,997 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:21,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:21,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:21,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:21,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:21,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:21,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,000 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:21,999 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,000 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:22,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:22,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,078 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/bac625162af44b92ab5ba93a075f07e2 2023-03-24 21:55:22,100 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:22,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:22,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:22,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:22,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:22,103 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:22,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:22,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:22,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:22,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,106 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:22,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:22,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:22,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:22,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:22,107 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:22,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:22,109 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,109 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:22,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59f9e279[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:22,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:22,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:22,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:22,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:22,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:22,114 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,114 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:22,114 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,114 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:22,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:22,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:22,116 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:22,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:22,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:22,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:22,116 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:22,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:22,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:22,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,130 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:22,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,130 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:22,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,231 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:22,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:22,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:22,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:22,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:22,234 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:22,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:22,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:22,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,236 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:22,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:22,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:22,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:22,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:22,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:22,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:22,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:22,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:22,238 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:22,238 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:22,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:22,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:22,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:22,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:22,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:22,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25014d80[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:22,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:22,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,243 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:22,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:22,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:22,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:22,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:22,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,245 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:22,247 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:22,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:22,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:22,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:22,247 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:22,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,253 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:22,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,253 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:22,254 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:22,354 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:22,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:22,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:22,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:22,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:22,357 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:22,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:22,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:22,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,359 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:22,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:22,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:22,361 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:22,361 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:22,361 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,361 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:22,361 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:22,361 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:22,361 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:22,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,363 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:22,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:22,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@530c4222[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:22,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:22,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,366 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,366 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:22,366 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:22,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:22,367 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:22,368 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,368 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:22,368 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,368 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:22,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:22,371 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:22,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:22,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:22,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:22,371 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:22,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,374 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:22,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,378 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:22,378 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694982409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694982411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694982416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,478 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:22,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:22,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:22,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:22,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:22,481 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:22,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:22,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:22,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,483 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,483 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,483 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,483 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:22,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:22,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:22,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:22,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:22,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,484 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:22,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:22,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:22,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:22,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:22,486 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,486 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:22,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=163 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/51a101f77a5443aea1359be0fd138462 2023-03-24 21:55:22,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:22,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b73eab[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:22,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:22,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,491 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/3b70ee8f1d1745e2aa1c168355f39a8e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e 2023-03-24 21:55:22,492 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:22,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:22,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:22,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:22,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:22,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e, entries=200, sequenceid=163, filesize=32.1 K 2023-03-24 21:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,496 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/bac625162af44b92ab5ba93a075f07e2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2 2023-03-24 21:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:22,497 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:22,497 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:22,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:22,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:22,498 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:22,499 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,499 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,499 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:22,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:22,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:22,501 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2, entries=150, sequenceid=163, filesize=11.8 K 2023-03-24 21:55:22,503 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:22,503 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/51a101f77a5443aea1359be0fd138462 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462 2023-03-24 21:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462, entries=150, sequenceid=163, filesize=11.8 K 2023-03-24 21:55:22,509 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3b6e2d0df6ef7b528783b27726388c56 in 1258ms, sequenceid=163, compaction requested=true 2023-03-24 21:55:22,509 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:22,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:22,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:22,509 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:22,509 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:22,509 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:22,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:22,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:22,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:22,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:22,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:22,511 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:22,511 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:22,511 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36164 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:22,511 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:22,511 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,511 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/3a0d88b4557e450bab8b12c4d76075a1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=35.3 K 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84734 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:22,512 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,512 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/8945ec32c21d4ace80c087ad68568d78, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=82.7 K 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3a0d88b4557e450bab8b12c4d76075a1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679694919405 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8945ec32c21d4ace80c087ad68568d78, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679694919405 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5f76b6c5982a451d880a52c8d52de902, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679694919488 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e9f4b53dbb914c06a7099f168539fd4e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679694919488 2023-03-24 21:55:22,512 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bac625162af44b92ab5ba93a075f07e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:22,513 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 3b70ee8f1d1745e2aa1c168355f39a8e, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:22,520 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#332 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:22,522 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#333 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:22,534 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/adf3ed932927414e8dfd80c0d76aee1d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/adf3ed932927414e8dfd80c0d76aee1d 2023-03-24 21:55:22,538 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into adf3ed932927414e8dfd80c0d76aee1d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:22,538 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:22,538 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=13, startTime=1679694922509; duration=0sec 2023-03-24 21:55:22,538 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:22,538 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:22,539 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47982 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:22,539 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/C is initiating minor compaction (all files) 2023-03-24 21:55:22,539 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/C in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,539 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/381fc0a1851040619756e1c1edd76cd0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=46.9 K 2023-03-24 21:55:22,540 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 381fc0a1851040619756e1c1edd76cd0, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1679694917690 2023-03-24 21:55:22,540 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting fefab3a6b1cc4af9a47a10bc336b025f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=125, earliestPutTs=1679694919405 2023-03-24 21:55:22,541 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6cf6b2593e564195ba8536731fd8ce6e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1679694919488 2023-03-24 21:55:22,541 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 51a101f77a5443aea1359be0fd138462, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:22,550 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#C#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:22,611 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:22,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:22,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:22,614 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:22,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:22,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:22,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:22,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:22,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:22,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:22,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:22,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:22,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:22,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:22,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:22,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:22,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:22,617 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:22,617 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:22,617 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:22,618 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:22,958 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/5ba8f1bcf5f2492587a8743f58f44252 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5ba8f1bcf5f2492587a8743f58f44252 2023-03-24 21:55:22,962 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/C of 3b6e2d0df6ef7b528783b27726388c56 into 5ba8f1bcf5f2492587a8743f58f44252(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:22,962 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:22,962 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/C, priority=12, startTime=1679694922509; duration=0sec 2023-03-24 21:55:22,962 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:23,030 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032478355d8e17a24362b94c71eaa99b4753 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032478355d8e17a24362b94c71eaa99b4753 2023-03-24 21:55:23,354 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/f39765cc3ba340e0a00e863e172602ec as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/f39765cc3ba340e0a00e863e172602ec 2023-03-24 21:55:23,360 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into f39765cc3ba340e0a00e863e172602ec(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:23,360 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:23,360 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=13, startTime=1679694922509; duration=0sec 2023-03-24 21:55:23,361 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:23,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:23,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:23,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694983437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694983440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,444 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=183, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/af081023a9bb4822a54f136902ce21ee 2023-03-24 21:55:23,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694983441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694983544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,550 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694983545, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,551 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694983548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694983752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694983752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,755 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:23,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694983752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:23,867 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/99d4bb5a65d74cddaa1e323854e7fffd 2023-03-24 21:55:24,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694984056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694984056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694984058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,276 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/168a3b2eb40f4002b74ddc9a4102ad5c 2023-03-24 21:55:24,283 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/af081023a9bb4822a54f136902ce21ee as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee 2023-03-24 21:55:24,287 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee, entries=150, sequenceid=183, filesize=25.3 K 2023-03-24 21:55:24,288 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/99d4bb5a65d74cddaa1e323854e7fffd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd 2023-03-24 21:55:24,296 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd, entries=150, sequenceid=183, filesize=11.8 K 2023-03-24 21:55:24,297 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/168a3b2eb40f4002b74ddc9a4102ad5c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c 2023-03-24 21:55:24,300 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c, entries=150, sequenceid=183, filesize=11.8 K 2023-03-24 21:55:24,302 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 3b6e2d0df6ef7b528783b27726388c56 in 1685ms, sequenceid=183, compaction requested=false 2023-03-24 21:55:24,302 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:24,302 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:24,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:24,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:24,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:24,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:24,304 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:24,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:24,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:24,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:24,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@334bfa4e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:24,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:24,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:24,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:24,309 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:24,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:24,311 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:24,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:24,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:24,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:24,311 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:24,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:24,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:24,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:24,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:24,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:24,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:24,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:24,317 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:24,317 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,417 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:24,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:24,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:24,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:24,420 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:24,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:24,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:24,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:24,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:24,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:24,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:24,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:24,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:24,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:24,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:24,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:24,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:24,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:24,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:24,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:24,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:24,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:24,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:24,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:24,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:24,425 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 21:55:24,425 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:24,426 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:24,426 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:24,426 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:24,426 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:24,426 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:24,437 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248dcaba8859f844d78adb6f8abc7d1f57 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248dcaba8859f844d78adb6f8abc7d1f57 2023-03-24 21:55:24,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:24,524 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:24,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694984576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694984578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694984578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694984579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694984685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694984686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694984686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694984686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,846 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/bc78da9f94a1462c9707e293dc049940 2023-03-24 21:55:24,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694984890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694984891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694984891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:24,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:24,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694984893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694985195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,198 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694985195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,200 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694985197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694985198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,257 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/a65ef73393b04e5590ab5529d28117c9 2023-03-24 21:55:25,667 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/d796e15018fa4fad910344c36423bf95 2023-03-24 21:55:25,671 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/bc78da9f94a1462c9707e293dc049940 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940 2023-03-24 21:55:25,674 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940, entries=150, sequenceid=198, filesize=25.3 K 2023-03-24 21:55:25,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/a65ef73393b04e5590ab5529d28117c9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9 2023-03-24 21:55:25,678 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:55:25,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/d796e15018fa4fad910344c36423bf95 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95 2023-03-24 21:55:25,682 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:55:25,682 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 3b6e2d0df6ef7b528783b27726388c56 in 1257ms, sequenceid=198, compaction requested=true 2023-03-24 21:55:25,682 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:25,682 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:25,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:25,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:25,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:25,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:25,684 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,684 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:25,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c09dcea[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:25,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:25,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:25,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:25,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,689 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:25,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:25,692 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:25,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:25,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:25,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:25,692 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:25,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:25,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:25,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,697 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,697 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:25,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:25,708 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:25,708 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:25,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694985719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694985720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694985721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694985722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,797 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:25,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:25,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:25,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:25,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:25,814 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:25,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:25,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:25,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:25,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,817 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:25,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:25,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:25,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:25,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:25,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:25,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:25,819 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:25,819 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:25,819 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:25,819 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:25,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:25,820 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:25,820 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,820 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:25,820 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:25,821 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:25,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d25901b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:25,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:25,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:25,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:25,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:25,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:25,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:25,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:25,828 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:25,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:25,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:25,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:25,828 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:25,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,830 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:25,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694985828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694985830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694985830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,837 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694985830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:25,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,838 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,838 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,938 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:25,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:25,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36940 deadline: 1679694985936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:25,942 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:25,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:25,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:25,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:25,944 DEBUG [Thread-2374] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8202 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:25,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,946 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,946 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:25,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:25,946 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,946 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:25,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,946 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,947 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:25,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:25,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:25,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:25,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:25,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:25,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:25,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:25,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:25,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:25,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:25,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:25,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:25,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:25,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:25,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:25,949 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:25,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:25,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31fbb172[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:25,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:25,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,952 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:25,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:25,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:25,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:25,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:25,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:25,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:25,956 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:25,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:25,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:25,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:25,957 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:25,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:25,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:25,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:25,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:25,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:25,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:25,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:25,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:25,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:25,973 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:25,973 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:25,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:25,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:25,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:25,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:25,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:25,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694986037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694986037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694986037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,043 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694986039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,073 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,075 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,076 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,077 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,077 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,077 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,077 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,077 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,078 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,079 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,080 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,080 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,080 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,080 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d0eb489[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,084 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,085 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,086 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,086 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,087 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,087 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,089 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,089 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,091 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,091 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,091 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,096 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,097 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,097 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,124 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e9bdd51da25841daaedeed6e309d650f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9bdd51da25841daaedeed6e309d650f 2023-03-24 21:55:26,197 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,200 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,200 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,202 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,202 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,202 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,204 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,204 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,204 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,204 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,206 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,206 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,207 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,207 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fb2cd4e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,210 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,210 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,214 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,214 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,221 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,222 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,222 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,322 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,324 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,325 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,327 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,328 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,328 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,329 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,329 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,329 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,329 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,329 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,334 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e1631eb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,336 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,336 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,337 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,337 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,340 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,340 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,340 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,340 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,340 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,343 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694986343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694986344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,345 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694986344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694986344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,346 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,351 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,351 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,351 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,351 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,351 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,352 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,352 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,452 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,455 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,458 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,458 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,458 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,459 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,459 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@620a717a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,463 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,463 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,463 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,463 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,463 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,463 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,466 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,466 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,466 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,469 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,469 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,471 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,472 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,474 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,475 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,475 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,479 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,530 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=218, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/65e8fa87c6bd4f05913b11fb97efeb45 2023-03-24 21:55:26,575 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,578 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,579 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,581 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,581 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,582 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,582 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,582 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,582 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,586 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a083b4e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,588 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,592 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,592 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,593 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,594 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,595 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,600 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,600 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,599 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,700 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,703 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,706 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,707 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,707 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53c957fc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,710 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,710 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,719 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,719 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,721 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,722 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,724 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,724 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,724 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,824 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,826 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,828 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,830 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,830 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,831 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,831 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,831 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,831 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,833 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,833 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,833 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,833 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c6fad07[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,836 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,843 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,843 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,851 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,851 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694986847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694986847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,852 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694986848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:26,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694986849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/a6dc51799de2436e98bd5492202e50a0 2023-03-24 21:55:26,951 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:26,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:26,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:26,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:26,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:26,954 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:26,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:26,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:26,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,956 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:26,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:26,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:26,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:26,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:26,957 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:26,958 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,958 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:26,958 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:26,958 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:26,958 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:26,961 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,961 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:26,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d5f8051[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:26,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:26,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:26,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:26,965 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:26,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:26,966 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:26,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:26,970 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:26,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:26,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:26,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:26,970 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:26,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,972 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:26,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:26,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:26,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:26,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:26,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:26,976 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:26,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:26,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:26,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:26,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:26,977 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:26,977 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:26,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:26,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:26,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:26,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:26,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,077 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:27,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:27,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:27,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:27,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:27,080 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:27,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:27,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:27,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:27,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:27,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:27,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:27,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:27,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:27,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:27,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:27,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:27,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:27,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:27,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:27,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:27,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,085 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,085 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:27,087 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@568602ff[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:27,087 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:27,087 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:27,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:27,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:27,088 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:27,091 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,091 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:27,091 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,091 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:27,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:27,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:27,093 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:27,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:27,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:27,093 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:27,093 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:27,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:27,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:27,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:27,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,102 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:27,102 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:27,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,202 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:27,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:27,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:27,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:27,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:27,206 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:27,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:27,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:27,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,208 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,209 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,209 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,209 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,210 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:27,210 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:27,210 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:27,210 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,210 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:27,210 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:27,210 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,210 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:27,211 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:27,211 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,211 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:27,211 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:27,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,212 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:27,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,213 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:27,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fe9d964[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:27,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:27,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:27,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:27,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:27,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:27,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:27,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,217 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:27,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:27,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:27,220 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:27,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:27,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:27,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:27,220 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:27,222 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,223 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,223 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,223 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,225 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:27,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,228 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:27,228 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:27,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,228 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:27,328 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:27,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:27,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:27,332 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:27,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:27,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:27,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,333 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,333 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,334 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,334 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,334 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,335 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:27,335 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:27,335 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:27,335 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,335 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:27,335 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:27,335 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:27,335 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,336 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:27,336 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:27,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,337 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,337 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,337 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:27,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eff520b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:27,338 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:27,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,341 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:27,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:27,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:27,341 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,341 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:27,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:27,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:27,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:27,345 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:27,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:27,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:27,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:27,345 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:27,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,347 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,347 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,348 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,350 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=218 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/7061e9405d3a4b519de93cae19551ea8 2023-03-24 21:55:27,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:27,352 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:27,353 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:27,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:27,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,353 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:27,355 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/65e8fa87c6bd4f05913b11fb97efeb45 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45 2023-03-24 21:55:27,358 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45, entries=200, sequenceid=218, filesize=32.1 K 2023-03-24 21:55:27,359 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/a6dc51799de2436e98bd5492202e50a0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0 2023-03-24 21:55:27,362 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0, entries=150, sequenceid=218, filesize=11.8 K 2023-03-24 21:55:27,362 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/7061e9405d3a4b519de93cae19551ea8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8 2023-03-24 21:55:27,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8, entries=150, sequenceid=218, filesize=11.8 K 2023-03-24 21:55:27,366 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 3b6e2d0df6ef7b528783b27726388c56 in 1658ms, sequenceid=218, compaction requested=true 2023-03-24 21:55:27,366 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:27,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:27,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:27,366 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:27,366 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:27,366 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:27,367 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:27,367 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110882 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:27,368 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:27,368 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:27,368 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,368 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,368 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/adf3ed932927414e8dfd80c0d76aee1d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=47.3 K 2023-03-24 21:55:27,368 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/f39765cc3ba340e0a00e863e172602ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=108.3 K 2023-03-24 21:55:27,368 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting adf3ed932927414e8dfd80c0d76aee1d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:27,368 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f39765cc3ba340e0a00e863e172602ec, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 99d4bb5a65d74cddaa1e323854e7fffd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679694921274 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting af081023a9bb4822a54f136902ce21ee, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679694921274 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a65ef73393b04e5590ab5529d28117c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694923436 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting bc78da9f94a1462c9707e293dc049940, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694923436 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a6dc51799de2436e98bd5492202e50a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:27,369 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 65e8fa87c6bd4f05913b11fb97efeb45, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:27,378 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#344 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:27,380 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#345 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:27,453 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:27,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:27,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:27,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:27,457 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:27,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:27,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:27,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:27,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:27,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:27,461 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:27,461 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:27,461 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:27,461 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:27,461 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:27,461 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,461 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:27,461 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:55:27,461 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:27,461 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:27,462 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:27,462 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:27,462 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:27,462 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:27,475 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324cb07a9f96369456cafb21268793838dc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cb07a9f96369456cafb21268793838dc 2023-03-24 21:55:27,789 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/1e692b2e1f554f5e9031dd4c4e9d2356 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1e692b2e1f554f5e9031dd4c4e9d2356 2023-03-24 21:55:27,793 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into 1e692b2e1f554f5e9031dd4c4e9d2356(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:27,793 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:27,793 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=12, startTime=1679694927366; duration=0sec 2023-03-24 21:55:27,793 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:27,793 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:27,794 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:27,794 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/C is initiating minor compaction (all files) 2023-03-24 21:55:27,795 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/C in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,795 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5ba8f1bcf5f2492587a8743f58f44252, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=47.3 K 2023-03-24 21:55:27,795 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5ba8f1bcf5f2492587a8743f58f44252, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=163, earliestPutTs=1679694920122 2023-03-24 21:55:27,796 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 168a3b2eb40f4002b74ddc9a4102ad5c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679694921274 2023-03-24 21:55:27,796 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d796e15018fa4fad910344c36423bf95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694923436 2023-03-24 21:55:27,796 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7061e9405d3a4b519de93cae19551ea8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:27,804 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#C#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:27,815 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/df9257605f9f4c4198813187ee1bf437 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/df9257605f9f4c4198813187ee1bf437 2023-03-24 21:55:27,821 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/C of 3b6e2d0df6ef7b528783b27726388c56 into df9257605f9f4c4198813187ee1bf437(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:27,821 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:27,821 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/C, priority=12, startTime=1679694927366; duration=0sec 2023-03-24 21:55:27,821 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:27,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:27,858 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:27,883 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=230, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/b2d960675d0145f8b43ee406b5cfdd7e 2023-03-24 21:55:27,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694987876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694987881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:27,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694987882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:27,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694987886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,897 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/ed2cef67e266403d97c9dcb7b41ecb50 2023-03-24 21:55:27,908 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=230 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/53bc5ff2c42749f9b4f7e863bb3099d4 2023-03-24 21:55:27,915 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/b2d960675d0145f8b43ee406b5cfdd7e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e 2023-03-24 21:55:27,919 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e, entries=150, sequenceid=230, filesize=25.3 K 2023-03-24 21:55:27,920 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/ed2cef67e266403d97c9dcb7b41ecb50 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50 2023-03-24 21:55:27,925 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50, entries=150, sequenceid=230, filesize=11.8 K 2023-03-24 21:55:27,926 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/53bc5ff2c42749f9b4f7e863bb3099d4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4 2023-03-24 21:55:27,930 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4, entries=150, sequenceid=230, filesize=11.8 K 2023-03-24 21:55:27,931 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 3b6e2d0df6ef7b528783b27726388c56 in 470ms, sequenceid=230, compaction requested=false 2023-03-24 21:55:27,932 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:27,932 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:27,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:27,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:27,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,935 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:27,935 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:27,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,936 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,937 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,937 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:27,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f831c8d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:27,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:27,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,940 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:27,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:27,940 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:27,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:27,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:27,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:27,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:27,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:27,945 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:27,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:27,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:27,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:27,945 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:27,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,947 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:27,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,949 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:27,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:27,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:27,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:27,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:27,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:27,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:27,955 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:27,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:27,955 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:27,955 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:27,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:27,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694927875, max timestamp=1679694927989Num uniques 859; 2023-03-24 21:55:27,991 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:27,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694927875, max timestamp=1679694927989Num uniques 859; 2023-03-24 21:55:27,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694927875, max timestamp=1679694927989Num uniques 859; 2023-03-24 21:55:27,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:27,992 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:27,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=26.84 KB heapSize=66.23 KB 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:27,993 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:28,055 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694928049, max timestamp=1679694928056Num uniques 859; 2023-03-24 21:55:28,058 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:28,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,059 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,059 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,059 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,059 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,059 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694928049, max timestamp=1679694928056Num uniques 859; 2023-03-24 21:55:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:28,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694928049, max timestamp=1679694928056Num uniques 859; 2023-03-24 21:55:28,060 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:28,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,063 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,063 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,063 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,063 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,063 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,064 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,064 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,064 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,064 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,066 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,066 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,067 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,067 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,069 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19f46878[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,070 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,070 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,071 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,071 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,071 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,074 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,074 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,074 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,074 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,074 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,077 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,077 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,077 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,082 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,083 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,083 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:28,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679694928080, max timestamp=1679694928101Num uniques 1018; 2023-03-24 21:55:28,103 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-24 21:55:28,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679694928080, max timestamp=1679694928101Num uniques 1018; 2023-03-24 21:55:28,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-24 21:55:28,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1679694928080, max timestamp=1679694928101Num uniques 1018; 2023-03-24 21:55:28,104 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-03-24 21:55:28,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694988131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694988137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694988137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694988138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,183 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,186 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,189 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,189 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1857b0db[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,200 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/2ec22c0a1c3d47e6b21b16742e71b83c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2ec22c0a1c3d47e6b21b16742e71b83c 2023-03-24 21:55:28,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,201 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,201 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,201 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,202 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,204 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into 2ec22c0a1c3d47e6b21b16742e71b83c(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:28,204 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:28,204 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=12, startTime=1679694927366; duration=0sec 2023-03-24 21:55:28,205 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,206 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,206 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,208 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,208 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,208 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,209 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,212 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,212 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,213 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:28,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,246 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694988239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694988247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694988248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,254 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694988248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,313 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,316 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,321 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,323 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c1cc6d0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,323 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,323 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,325 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,326 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,326 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,327 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,327 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,329 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,329 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,329 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,329 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,329 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:28,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,338 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,338 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,412 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032428ce9e48b81e4af9b4f6989cee001fc6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428ce9e48b81e4af9b4f6989cee001fc6 2023-03-24 21:55:28,438 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,441 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b9cc2aa[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694988448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,454 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694988449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,455 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,455 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,462 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,462 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694988455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37004 deadline: 1679694988455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,562 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,565 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,569 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,570 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6191d609[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,577 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,581 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,581 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,587 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,587 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,588 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,588 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,688 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,691 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,697 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,697 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,697 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44657388[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,702 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,704 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,704 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:28,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,713 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,714 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,714 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694988755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,761 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37012 deadline: 1679694988756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694988765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,771 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:28,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:36954 deadline: 1679694988765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,814 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,817 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,817 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,817 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=254, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/50d9fbb14f5f4dd1a37070f62fed15c9 2023-03-24 21:55:28,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,821 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,821 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e150792[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,829 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,831 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,831 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,833 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,833 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,840 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,840 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,940 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:28,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:28,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:28,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:28,943 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:28,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:28,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:28,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,945 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,945 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:28,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,946 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:28,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:28,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:28,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:28,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:28,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:28,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:28,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:28,947 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:28,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:28,949 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:28,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23210f68[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:28,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:28,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,952 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:28,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:28,953 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:28,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,953 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:28,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:28,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:28,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:28,957 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:28,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:28,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:28,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:28,957 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:28,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:28,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,959 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:28,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:28,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:28,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:28,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:28,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:28,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:28,965 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:28,965 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:28,966 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,066 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,069 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,070 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,072 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,073 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,073 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,073 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,073 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,073 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,073 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,073 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,074 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:29,074 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,076 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,076 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,076 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,077 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6fb49b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,080 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,083 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,085 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,085 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,085 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,085 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,085 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,087 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,090 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,091 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,090 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,191 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,195 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,203 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,203 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,203 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:29,203 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,203 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,204 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,205 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,205 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237977ad[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,219 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,219 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,220 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,220 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,221 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,222 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,226 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,227 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,227 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,226 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/9637fcf51e084751a61b23d07b182559 2023-03-24 21:55:29,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:37026 deadline: 1679694989264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694989265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:29,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694989274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:29,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694989276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,332 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,335 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,337 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,337 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,338 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,338 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:29,340 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,340 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1566ddc7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,343 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,344 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,344 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,345 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,345 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,345 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,347 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,350 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,350 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,350 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,352 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,354 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,356 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,357 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,357 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,357 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,457 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,460 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,463 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,463 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,464 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,464 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,464 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:29,464 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,464 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e17da4c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,469 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,470 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,471 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,471 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,474 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,474 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,477 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,478 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,481 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,482 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,482 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,582 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,586 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,588 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,589 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,589 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,589 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,589 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,589 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,589 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,589 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:29,590 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,590 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fed6a79[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,593 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,596 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,601 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,608 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,614 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,614 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,614 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,620 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,621 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,621 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,621 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:29,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=254 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/dc2b1314534e4bd4968bbbbfc437d861 2023-03-24 21:55:29,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/50d9fbb14f5f4dd1a37070f62fed15c9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9 2023-03-24 21:55:29,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9, entries=250, sequenceid=254, filesize=38.9 K 2023-03-24 21:55:29,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/9637fcf51e084751a61b23d07b182559 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559 2023-03-24 21:55:29,653 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 21:55:29,654 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/dc2b1314534e4bd4968bbbbfc437d861 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861 2023-03-24 21:55:29,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861, entries=150, sequenceid=254, filesize=11.8 K 2023-03-24 21:55:29,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=161.02 KB/164880 for 3b6e2d0df6ef7b528783b27726388c56 in 1665ms, sequenceid=254, compaction requested=true 2023-03-24 21:55:29,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:29,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:29,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:29,658 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:29,658 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:29,658 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:29,659 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:29,659 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 92116 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:29,659 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:29,659 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:29,659 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,659 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,659 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1e692b2e1f554f5e9031dd4c4e9d2356, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=35.7 K 2023-03-24 21:55:29,659 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2ec22c0a1c3d47e6b21b16742e71b83c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=90.0 K 2023-03-24 21:55:29,659 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1e692b2e1f554f5e9031dd4c4e9d2356, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:29,660 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2ec22c0a1c3d47e6b21b16742e71b83c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:29,660 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ed2cef67e266403d97c9dcb7b41ecb50, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679694925719 2023-03-24 21:55:29,661 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b2d960675d0145f8b43ee406b5cfdd7e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679694925719 2023-03-24 21:55:29,661 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9637fcf51e084751a61b23d07b182559, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927880 2023-03-24 21:55:29,661 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 50d9fbb14f5f4dd1a37070f62fed15c9, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927875 2023-03-24 21:55:29,669 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#354 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:29,678 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#353 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:29,688 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/1a431861aebb4b1cb84454e0b42a45c9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a431861aebb4b1cb84454e0b42a45c9 2023-03-24 21:55:29,692 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into 1a431861aebb4b1cb84454e0b42a45c9(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:29,692 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:29,692 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=13, startTime=1679694929658; duration=0sec 2023-03-24 21:55:29,692 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:29,692 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:29,693 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:29,693 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/C is initiating minor compaction (all files) 2023-03-24 21:55:29,693 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/C in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,693 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/df9257605f9f4c4198813187ee1bf437, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=35.7 K 2023-03-24 21:55:29,694 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting df9257605f9f4c4198813187ee1bf437, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=218, earliestPutTs=1679694924576 2023-03-24 21:55:29,694 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 53bc5ff2c42749f9b4f7e863bb3099d4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=230, earliestPutTs=1679694925719 2023-03-24 21:55:29,695 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting dc2b1314534e4bd4968bbbbfc437d861, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927880 2023-03-24 21:55:29,700 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#C#compaction#355 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:29,721 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,724 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,724 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,727 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,727 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,728 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,729 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,729 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,729 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,729 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=161.02 KB heapSize=445.08 KB 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:29,730 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:29,741 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032411be30e839ee473fa362a2be47b54715 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032411be30e839ee473fa362a2be47b54715 2023-03-24 21:55:29,747 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/0e4b28d287b649959b4c2afa0429f1b8 2023-03-24 21:55:29,761 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/85638c8f457244bb9fcd857f0a286010 2023-03-24 21:55:29,776 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e 2023-03-24 21:55:29,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/0e4b28d287b649959b4c2afa0429f1b8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8 2023-03-24 21:55:29,784 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8, entries=150, sequenceid=310, filesize=25.4 K 2023-03-24 21:55:29,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/85638c8f457244bb9fcd857f0a286010 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010 2023-03-24 21:55:29,789 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010, entries=150, sequenceid=310, filesize=11.9 K 2023-03-24 21:55:29,789 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e 2023-03-24 21:55:29,792 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e, entries=150, sequenceid=310, filesize=11.9 K 2023-03-24 21:55:29,793 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~445.03 KB/455712, currentSize=0 B/0 for 3b6e2d0df6ef7b528783b27726388c56 in 64ms, sequenceid=310, compaction requested=false 2023-03-24 21:55:29,793 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:29,793 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,794 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,795 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6741a036[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,798 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,799 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,799 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,803 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,805 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,805 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,807 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,807 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,807 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,808 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,809 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:29,811 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,811 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,811 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,912 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:29,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:29,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:29,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:29,916 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:29,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:29,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:29,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:29,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:29,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:29,919 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:29,920 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,920 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:29,920 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:29,920 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:29,920 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:29,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:29,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446ad1fc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:29,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:29,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,926 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,926 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:29,926 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:29,926 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:29,926 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,926 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:29,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,927 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:29,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,928 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:29,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:29,930 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:29,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:29,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:29,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:29,930 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:29,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:29,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:29,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:29,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:29,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:29,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:29,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:29,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:29,938 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:29,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:29,939 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:29,939 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:29,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:29,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:29,938 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:29,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:29,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:29,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,039 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:30,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:30,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:30,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:30,042 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:30,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:30,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:30,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,044 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:30,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:30,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,045 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:30,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:30,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:30,047 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:30,047 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:30,047 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:30,047 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:30,047 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:30,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:30,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44afa687[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:30,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:30,051 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,052 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:30,053 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:30,053 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:30,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:30,059 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:30,059 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:30,059 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:30,062 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:30,062 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:30,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:30,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:30,062 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:30,062 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:30,064 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,064 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,064 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:30,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,066 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:30,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:30,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:30,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:30,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:30,071 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:30,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:30,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:30,071 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:30,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,072 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:30,072 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,072 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,108 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/ab01184fece94fb39e426f98250865f5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ab01184fece94fb39e426f98250865f5 2023-03-24 21:55:30,113 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/C of 3b6e2d0df6ef7b528783b27726388c56 into ab01184fece94fb39e426f98250865f5(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:30,113 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:30,113 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/C, priority=13, startTime=1679694929658; duration=0sec 2023-03-24 21:55:30,113 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:30,172 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:30,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:30,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:30,176 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:30,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:30,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:30,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:30,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:30,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,182 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,182 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:30,183 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:30,183 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:30,183 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:30,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:30,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,186 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:30,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:30,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:30,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:30,190 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:30,190 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:30,192 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:30,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f5b2c9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:30,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:30,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,197 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:30,197 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:30,197 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:30,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:30,199 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:30,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:30,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:30,202 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:30,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:30,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:30,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:30,203 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:30,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,204 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:30,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:30,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:30,205 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:30,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:30,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,206 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:30,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:30,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:30,209 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:30,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:30,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:30,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:30,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,210 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:30,210 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,210 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:30,210 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:30,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:30,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:30,210 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:30,210 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,310 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:30,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:30,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:30,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:30,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:30,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:30,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:30,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:30,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:30,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:30,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:30,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:30,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:30,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:30,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:30,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:30,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:30,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:30,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:30,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:30,320 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:30,320 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:30,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:30,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694930371, max timestamp=1679694930381Num uniques 859; 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694930371, max timestamp=1679694930381Num uniques 859; 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694930371, max timestamp=1679694930381Num uniques 859; 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:30,383 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:30,408 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694930399, max timestamp=1679694930406Num uniques 1118; 2023-03-24 21:55:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694930399, max timestamp=1679694930406Num uniques 1118; 2023-03-24 21:55:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694930399, max timestamp=1679694930406Num uniques 1118; 2023-03-24 21:55:30,409 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694930430, max timestamp=1679694930442Num uniques 1127; 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694930430, max timestamp=1679694930442Num uniques 1127; 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694930430, max timestamp=1679694930442Num uniques 1127; 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:55:30,445 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=8102, suffix heap size=145176,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=177 2023-03-24 21:55:30,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694930469, max timestamp=1679694930486Num uniques 1036; 2023-03-24 21:55:30,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:55:30,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694930469, max timestamp=1679694930486Num uniques 1036; 2023-03-24 21:55:30,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:55:30,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=286, cellSize=13096, totalHeapSize=29984, min timestamp=1679694930469, max timestamp=1679694930486Num uniques 1036; 2023-03-24 21:55:30,490 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=13096, suffix heap size=135712,new segment heap size=29984 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=286 2023-03-24 21:55:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694930523, max timestamp=1679694930567Num uniques 1145; 2023-03-24 21:55:30,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-03-24 21:55:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694930523, max timestamp=1679694930567Num uniques 1145; 2023-03-24 21:55:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694930523, max timestamp=1679694930567Num uniques 1145; 2023-03-24 21:55:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-03-24 21:55:30,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52440, new segment data size=8930, suffix heap size=147048,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1145, new segment cells count=195 2023-03-24 21:55:30,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679694930589, max timestamp=1679694930609Num uniques 1054; 2023-03-24 21:55:30,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-24 21:55:30,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679694930589, max timestamp=1679694930609Num uniques 1054; 2023-03-24 21:55:30,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1679694930589, max timestamp=1679694930609Num uniques 1054; 2023-03-24 21:55:30,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-24 21:55:30,612 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=11630, suffix heap size=137584,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=254 2023-03-24 21:55:30,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:30,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694990644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694930627, max timestamp=1679694930648Num uniques 1113; 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694930627, max timestamp=1679694930648Num uniques 1113; 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694930627, max timestamp=1679694930648Num uniques 1113; 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 21:55:30,651 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694930668, max timestamp=1679694930688Num uniques 1072; 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694930668, max timestamp=1679694930688Num uniques 1072; 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=222, cellSize=10162, totalHeapSize=23328, min timestamp=1679694930668, max timestamp=1679694930688Num uniques 1072; 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 21:55:30,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=10162, suffix heap size=139456,new segment heap size=23328 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=222 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679694930712, max timestamp=1679694930727Num uniques 1081; 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679694930712, max timestamp=1679694930727Num uniques 1081; 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679694930712, max timestamp=1679694930727Num uniques 1081; 2023-03-24 21:55:30,729 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49506, new segment data size=8286, suffix heap size=140392,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1081, new segment cells count=181 2023-03-24 21:55:30,732 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324d65ee39d02de4154bc1cfbbd3d70502a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d65ee39d02de4154bc1cfbbd3d70502a 2023-03-24 21:55:30,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694930747, max timestamp=1679694930764Num uniques 1040; 2023-03-24 21:55:30,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-24 21:55:30,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694930747, max timestamp=1679694930764Num uniques 1040; 2023-03-24 21:55:30,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694930747, max timestamp=1679694930764Num uniques 1040; 2023-03-24 21:55:30,767 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-24 21:55:30,768 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-03-24 21:55:30,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694930789, max timestamp=1679694930797Num uniques 1049; 2023-03-24 21:55:30,799 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-24 21:55:30,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694930789, max timestamp=1679694930797Num uniques 1049; 2023-03-24 21:55:30,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694930789, max timestamp=1679694930797Num uniques 1049; 2023-03-24 21:55:30,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-24 21:55:30,800 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=13694, suffix heap size=137064,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=299 2023-03-24 21:55:30,825 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:30,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694990822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:30,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694930816, max timestamp=1679694930824Num uniques 1158; 2023-03-24 21:55:30,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:30,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694930816, max timestamp=1679694930824Num uniques 1158; 2023-03-24 21:55:30,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:30,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694930816, max timestamp=1679694930824Num uniques 1158; 2023-03-24 21:55:30,827 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:30,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679694930863, max timestamp=1679694930877Num uniques 1117; 2023-03-24 21:55:30,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 21:55:30,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679694930863, max timestamp=1679694930877Num uniques 1117; 2023-03-24 21:55:30,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 21:55:30,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679694930863, max timestamp=1679694930877Num uniques 1117; 2023-03-24 21:55:30,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=12222, suffix heap size=144136,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=267 2023-03-24 21:55:30,901 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/ef91bca5bbfc44ce98a78829abc5f0be as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ef91bca5bbfc44ce98a78829abc5f0be 2023-03-24 21:55:30,904 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into ef91bca5bbfc44ce98a78829abc5f0be(size=25.8 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 21:55:30,904 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:30,904 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=13, startTime=1679694929658; duration=1sec 2023-03-24 21:55:30,904 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:30,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679694930904, max timestamp=1679694930917Num uniques 1126; 2023-03-24 21:55:30,919 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-24 21:55:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679694930904, max timestamp=1679694930917Num uniques 1126; 2023-03-24 21:55:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-24 21:55:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1679694930904, max timestamp=1679694930917Num uniques 1126; 2023-03-24 21:55:30,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51566, new segment data size=12636, suffix heap size=145072,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=276 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694930932, max timestamp=1679694930947Num uniques 1135; 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694930932, max timestamp=1679694930947Num uniques 1135; 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694930932, max timestamp=1679694930947Num uniques 1135; 2023-03-24 21:55:30,950 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=51980, new segment data size=10760, suffix heap size=146008,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1135, new segment cells count=235 2023-03-24 21:55:30,979 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13454, totalHeapSize=30816, min timestamp=1679694930969, max timestamp=1679694930977Num uniques 1094; 2023-03-24 21:55:30,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50094, new segment data size=13454, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-24 21:55:30,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1679694930969, max timestamp=1679694930977Num uniques 1094; 2023-03-24 21:55:30,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-24 21:55:30,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13454, totalHeapSize=30816, min timestamp=1679694930969, max timestamp=1679694930977Num uniques 1094; 2023-03-24 21:55:30,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50094, new segment data size=13454, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-24 21:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679694931002, max timestamp=1679694931015Num uniques 1153; 2023-03-24 21:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679694931002, max timestamp=1679694931015Num uniques 1153; 2023-03-24 21:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-24 21:55:31,017 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-24 21:55:31,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679694931002, max timestamp=1679694931015Num uniques 1153; 2023-03-24 21:55:31,018 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-24 21:55:31,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694931087, max timestamp=1679694931096Num uniques 1062; 2023-03-24 21:55:31,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-24 21:55:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694931087, max timestamp=1679694931096Num uniques 1062; 2023-03-24 21:55:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-24 21:55:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694931087, max timestamp=1679694931096Num uniques 1062; 2023-03-24 21:55:31,099 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694931112, max timestamp=1679694931124Num uniques 1021; 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694931112, max timestamp=1679694931124Num uniques 1021; 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694931112, max timestamp=1679694931124Num uniques 1021; 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:31,127 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:31,138 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/108bcd3b81fc4d108c7dc2e9040278d4 2023-03-24 21:55:31,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679694931147, max timestamp=1679694931155Num uniques 1080; 2023-03-24 21:55:31,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679694931147, max timestamp=1679694931155Num uniques 1080; 2023-03-24 21:55:31,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-24 21:55:31,157 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-24 21:55:31,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679694931147, max timestamp=1679694931155Num uniques 1080; 2023-03-24 21:55:31,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-24 21:55:31,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694931175, max timestamp=1679694931198Num uniques 1089; 2023-03-24 21:55:31,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-24 21:55:31,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694931175, max timestamp=1679694931198Num uniques 1089; 2023-03-24 21:55:31,200 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-24 21:55:31,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694931175, max timestamp=1679694931198Num uniques 1089; 2023-03-24 21:55:31,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=10944, suffix heap size=141224,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=239 2023-03-24 21:55:31,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1679694931221, max timestamp=1679694931228Num uniques 1098; 2023-03-24 21:55:31,230 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-24 21:55:31,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9068, totalHeapSize=20832, min timestamp=1679694931221, max timestamp=1679694931228Num uniques 1098; 2023-03-24 21:55:31,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=9068, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-24 21:55:31,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=198, cellSize=9066, totalHeapSize=20832, min timestamp=1679694931221, max timestamp=1679694931229Num uniques 1098; 2023-03-24 21:55:31,231 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50286, new segment data size=9066, suffix heap size=142160,new segment heap size=20832 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=198 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679694931260, max timestamp=1679694931269Num uniques 1057; 2023-03-24 21:55:31,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679694931260, max timestamp=1679694931269Num uniques 1057; 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=157, cellSize=7185, totalHeapSize=16568, min timestamp=1679694931260, max timestamp=1679694931269Num uniques 1057; 2023-03-24 21:55:31,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694991262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,271 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=48405, new segment data size=7185, suffix heap size=137896,new segment heap size=16568 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1057, new segment cells count=157 2023-03-24 21:55:31,326 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679694931296, max timestamp=1679694931324Num uniques 1016; 2023-03-24 21:55:31,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-03-24 21:55:31,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679694931296, max timestamp=1679694931324Num uniques 1016; 2023-03-24 21:55:31,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-03-24 21:55:31,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=216, cellSize=9886, totalHeapSize=22704, min timestamp=1679694931296, max timestamp=1679694931324Num uniques 1016; 2023-03-24 21:55:31,327 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46526, new segment data size=9886, suffix heap size=133632,new segment heap size=22704 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1016, new segment cells count=216 2023-03-24 21:55:31,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694931355, max timestamp=1679694931367Num uniques 1075; 2023-03-24 21:55:31,369 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-24 21:55:31,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694931355, max timestamp=1679694931367Num uniques 1075; 2023-03-24 21:55:31,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694931355, max timestamp=1679694931367Num uniques 1075; 2023-03-24 21:55:31,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-24 21:55:31,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=49230, new segment data size=10300, suffix heap size=139768,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1075, new segment cells count=225 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694931397, max timestamp=1679694931404Num uniques 1084; 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694931397, max timestamp=1679694931404Num uniques 1084; 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694931397, max timestamp=1679694931404Num uniques 1084; 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:31,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:31,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1679694931421, max timestamp=1679694931429Num uniques 1093; 2023-03-24 21:55:31,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-24 21:55:31,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1679694931421, max timestamp=1679694931429Num uniques 1093; 2023-03-24 21:55:31,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=293, cellSize=13418, totalHeapSize=30712, min timestamp=1679694931421, max timestamp=1679694931429Num uniques 1093; 2023-03-24 21:55:31,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-24 21:55:31,432 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=13418, suffix heap size=141640,new segment heap size=30712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=293 2023-03-24 21:55:31,459 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694991454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694991455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1679694931445, max timestamp=1679694931459Num uniques 1152; 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=13830, suffix heap size=147776,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=302 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1679694931445, max timestamp=1679694931459Num uniques 1152; 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=302, cellSize=13830, totalHeapSize=31648, min timestamp=1679694931445, max timestamp=1679694931459Num uniques 1152; 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=13830, suffix heap size=147776,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=302 2023-03-24 21:55:31,461 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52760, new segment data size=13830, suffix heap size=147776,new segment heap size=31648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1152, new segment cells count=302 2023-03-24 21:55:31,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1679694931479, max timestamp=1679694931518Num uniques 1161; 2023-03-24 21:55:31,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1679694931479, max timestamp=1679694931518Num uniques 1161; 2023-03-24 21:55:31,520 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=211, cellSize=9656, totalHeapSize=22184, min timestamp=1679694931479, max timestamp=1679694931518Num uniques 1161; 2023-03-24 21:55:31,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-03-24 21:55:31,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-03-24 21:55:31,521 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=53166, new segment data size=9656, suffix heap size=148712,new segment heap size=22184 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1161, new segment cells count=211 2023-03-24 21:55:31,547 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/43adf993033b4e21ba4d8ca0ed5e342d 2023-03-24 21:55:31,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679694931538, max timestamp=1679694931566Num uniques 1070; 2023-03-24 21:55:31,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-24 21:55:31,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679694931538, max timestamp=1679694931566Num uniques 1070; 2023-03-24 21:55:31,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=220, cellSize=10070, totalHeapSize=23120, min timestamp=1679694931538, max timestamp=1679694931566Num uniques 1070; 2023-03-24 21:55:31,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-24 21:55:31,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49000, new segment data size=10070, suffix heap size=139248,new segment heap size=23120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1070, new segment cells count=220 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1679694931579, max timestamp=1679694931601Num uniques 1079; 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1679694931579, max timestamp=1679694931601Num uniques 1079; 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=279, cellSize=12774, totalHeapSize=29256, min timestamp=1679694931579, max timestamp=1679694931601Num uniques 1079; 2023-03-24 21:55:31,604 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49414, new segment data size=12774, suffix heap size=140184,new segment heap size=29256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1079, new segment cells count=279 2023-03-24 21:55:31,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694991633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694931619, max timestamp=1679694931638Num uniques 1138; 2023-03-24 21:55:31,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=52118, new segment data size=8608, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-03-24 21:55:31,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694931619, max timestamp=1679694931638Num uniques 1138; 2023-03-24 21:55:31,640 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52118, new segment data size=8608, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-03-24 21:55:31,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=188, cellSize=8608, totalHeapSize=19792, min timestamp=1679694931619, max timestamp=1679694931638Num uniques 1138; 2023-03-24 21:55:31,641 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52118, new segment data size=8608, suffix heap size=146320,new segment heap size=19792 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1138, new segment cells count=188 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679694931670, max timestamp=1679694931681Num uniques 1047; 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=9022, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679694931670, max timestamp=1679694931681Num uniques 1047; 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=197, cellSize=9022, totalHeapSize=20728, min timestamp=1679694931670, max timestamp=1679694931681Num uniques 1047; 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=9022, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-03-24 21:55:31,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=47952, new segment data size=9022, suffix heap size=136856,new segment heap size=20728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1047, new segment cells count=197 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=306, cellSize=14010, totalHeapSize=32064, min timestamp=1679694931703, max timestamp=1679694931724Num uniques 1056; 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=306, cellSize=14010, totalHeapSize=32064, min timestamp=1679694931703, max timestamp=1679694931724Num uniques 1056; 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=14010, suffix heap size=137792,new segment heap size=32064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=306 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=14010, suffix heap size=137792,new segment heap size=32064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=306 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=306, cellSize=14010, totalHeapSize=32064, min timestamp=1679694931703, max timestamp=1679694931724Num uniques 1056; 2023-03-24 21:55:31,727 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48360, new segment data size=14010, suffix heap size=137792,new segment heap size=32064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1056, new segment cells count=306 2023-03-24 21:55:31,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694991782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694991782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1679694931769, max timestamp=1679694931788Num uniques 1165; 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=53350, new segment data size=9840, suffix heap size=149128,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1165, new segment cells count=215 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1679694931769, max timestamp=1679694931788Num uniques 1165; 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53350, new segment data size=9840, suffix heap size=149128,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1165, new segment cells count=215 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=215, cellSize=9840, totalHeapSize=22600, min timestamp=1679694931769, max timestamp=1679694931788Num uniques 1165; 2023-03-24 21:55:31,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=53350, new segment data size=9840, suffix heap size=149128,new segment heap size=22600 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1165, new segment cells count=215 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1679694931810, max timestamp=1679694931828Num uniques 1074; 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1679694931810, max timestamp=1679694931828Num uniques 1074; 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=274, cellSize=12544, totalHeapSize=28736, min timestamp=1679694931810, max timestamp=1679694931828Num uniques 1074; 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=12544, suffix heap size=139664,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=274 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=12544, suffix heap size=139664,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=274 2023-03-24 21:55:31,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=49184, new segment data size=12544, suffix heap size=139664,new segment heap size=28736 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1074, new segment cells count=274 2023-03-24 21:55:31,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679694931857, max timestamp=1679694931873Num uniques 1133; 2023-03-24 21:55:31,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679694931857, max timestamp=1679694931873Num uniques 1133; 2023-03-24 21:55:31,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=233, cellSize=10668, totalHeapSize=24472, min timestamp=1679694931857, max timestamp=1679694931873Num uniques 1133; 2023-03-24 21:55:31,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=10668, suffix heap size=145800,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=233 2023-03-24 21:55:31,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=10668, suffix heap size=145800,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=233 2023-03-24 21:55:31,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=51888, new segment data size=10668, suffix heap size=145800,new segment heap size=24472 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1133, new segment cells count=233 2023-03-24 21:55:31,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1679694931894, max timestamp=1679694931904Num uniques 1092; 2023-03-24 21:55:31,906 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=13372, suffix heap size=141536,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=292 2023-03-24 21:55:31,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1679694931894, max timestamp=1679694931904Num uniques 1092; 2023-03-24 21:55:31,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=13372, suffix heap size=141536,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=292 2023-03-24 21:55:31,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=292, cellSize=13372, totalHeapSize=30608, min timestamp=1679694931894, max timestamp=1679694931904Num uniques 1092; 2023-03-24 21:55:31,907 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50012, new segment data size=13372, suffix heap size=141536,new segment heap size=30608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1092, new segment cells count=292 2023-03-24 21:55:31,932 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694991926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:31,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694991929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1679694931921, max timestamp=1679694931931Num uniques 1151; 2023-03-24 21:55:31,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=11495, suffix heap size=147672,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=251 2023-03-24 21:55:31,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1679694931921, max timestamp=1679694931931Num uniques 1151; 2023-03-24 21:55:31,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=11495, suffix heap size=147672,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=251 2023-03-24 21:55:31,933 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=251, cellSize=11495, totalHeapSize=26344, min timestamp=1679694931921, max timestamp=1679694931931Num uniques 1151; 2023-03-24 21:55:31,934 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52715, new segment data size=11495, suffix heap size=147672,new segment heap size=26344 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1151, new segment cells count=251 2023-03-24 21:55:31,959 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/eb22d879f37145978bffe4dd7a7f35ef 2023-03-24 21:55:31,962 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/108bcd3b81fc4d108c7dc2e9040278d4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4 2023-03-24 21:55:31,965 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4, entries=150, sequenceid=320, filesize=25.4 K 2023-03-24 21:55:31,966 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/43adf993033b4e21ba4d8ca0ed5e342d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d 2023-03-24 21:55:31,969 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d, entries=100, sequenceid=320, filesize=9.5 K 2023-03-24 21:55:31,969 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/eb22d879f37145978bffe4dd7a7f35ef as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef 2023-03-24 21:55:31,972 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef, entries=100, sequenceid=320, filesize=9.5 K 2023-03-24 21:55:31,973 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=114.05 KB/116790 for 3b6e2d0df6ef7b528783b27726388c56 in 1653ms, sequenceid=320, compaction requested=true 2023-03-24 21:55:31,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:31,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:31,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:31,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:31,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:31,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:31,974 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=120.76 KB heapSize=336.52 KB 2023-03-24 21:55:31,975 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:31,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:31,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:31,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,976 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:31,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:31,976 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:31,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:31,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:31,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:31,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5ff6f1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:31,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:31,978 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:31,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:31,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:31,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:31,981 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:31,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:31,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:31,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:31,984 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:31,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:31,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:31,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:31,984 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:31,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:31,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:31,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,988 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:31,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:31,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:31,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:31,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:31,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:31,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:31,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:31,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:31,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:31,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,996 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:31,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:31,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:31,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:31,996 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:31,996 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694992006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694992007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694992034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694992034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,096 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,099 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,100 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,100 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,110 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58a82b77[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,114 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,115 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694992112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,120 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,120 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,122 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694992114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,131 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,132 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:32,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,232 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,235 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,237 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,238 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,239 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,241 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@610a257b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,243 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,244 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,244 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694992239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694992239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,246 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,253 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,258 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,258 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694992316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694992324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,358 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,362 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,369 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,370 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,370 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,370 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,370 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,370 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,371 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,372 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c64b069[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,376 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,376 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,378 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,378 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,381 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,382 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,382 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,387 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,387 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,387 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,387 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303248916492acdaa408a90eba96b2ee15165 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248916492acdaa408a90eba96b2ee15165 2023-03-24 21:55:32,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:32,387 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,387 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,487 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,490 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,491 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,492 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,493 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,493 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,493 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,494 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,494 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,494 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,494 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,494 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,494 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,494 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,496 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,496 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,496 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,496 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59392b6a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,500 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,500 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,501 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,501 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,501 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,502 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,505 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,505 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,505 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,505 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,505 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,511 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,511 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,550 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694992546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694992547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,611 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,615 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,615 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,621 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,623 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,624 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,625 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,625 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,626 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40d350a9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,627 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,627 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,628 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,629 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,630 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,632 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,632 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694992627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,633 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:32,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694992630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,640 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:32,640 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,740 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,743 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,744 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,744 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,746 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,746 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70c29435[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,750 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,753 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,756 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,757 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,758 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,758 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,758 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:32,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,764 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,764 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,792 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1008, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/7cb05120624641b1b4363772cd20ee75 2023-03-24 21:55:32,864 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,867 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,870 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,872 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,872 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,872 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,872 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,874 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:32,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@396c95bc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:32,876 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:32,876 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,878 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:32,878 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:32,878 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:32,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:32,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:32,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,879 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:32,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:32,883 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:32,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:32,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:32,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:32,883 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:32,886 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:32,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,887 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:32,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,889 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:32,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:32,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:32,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,890 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,890 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,890 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,890 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:32,890 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:32,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:32,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:32,990 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:32,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:32,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:32,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:32,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:32,994 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,994 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:32,995 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:32,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:32,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:32,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:32,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:32,997 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,997 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:32,997 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:32,997 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:32,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:32,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,999 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:32,999 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:32,999 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:32,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:33,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@291e9b80[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:33,001 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:33,001 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,006 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:33,006 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:33,006 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:33,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:33,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:33,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,008 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:33,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:33,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:33,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:33,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:33,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:33,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:33,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:33,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,013 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:33,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:33,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:33,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,017 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:33,017 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:33,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:33,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:36954 deadline: 1679694993055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:33,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37026 deadline: 1679694993055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,117 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:33,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:33,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:33,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:33,120 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:33,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:33,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:33,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:33,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:33,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:33,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:33,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:33,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:33,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:33,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:33,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:33,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:33,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:33,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,129 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,129 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:33,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7312c012[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:33,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:33,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,132 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:33,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:33,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:33,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:33,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:33,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:33,138 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:33,138 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:33,138 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:33,138 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:33,138 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:33,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,141 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,142 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694993136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:33,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37012 deadline: 1679694993136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:33,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,148 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:33,148 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:33,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:33,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1008 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/df8cfe79dcc94c4abdeaee695dc95962 2023-03-24 21:55:33,235 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1008 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/2df3c55cf7414a0a83f83566211c9b42 2023-03-24 21:55:33,240 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/7cb05120624641b1b4363772cd20ee75 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75 2023-03-24 21:55:33,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75, entries=250, sequenceid=1008, filesize=39.2 K 2023-03-24 21:55:33,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/df8cfe79dcc94c4abdeaee695dc95962 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962 2023-03-24 21:55:33,248 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:33,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:33,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:33,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:33,254 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:33,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:33,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:33,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962, entries=150, sequenceid=1008, filesize=11.9 K 2023-03-24 21:55:33,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:33,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/2df3c55cf7414a0a83f83566211c9b42 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42 2023-03-24 21:55:33,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:33,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:33,258 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:33,258 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:33,258 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:33,258 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,258 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:33,261 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42, entries=150, sequenceid=1008, filesize=11.9 K 2023-03-24 21:55:33,262 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~356.39 KB/364944, currentSize=53.67 KB/54960 for 3b6e2d0df6ef7b528783b27726388c56 in 1288ms, sequenceid=1008, compaction requested=true 2023-03-24 21:55:33,262 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:33,262 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:33,262 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:33,262 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:33,263 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:33,264 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118660 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:33,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:33,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,264 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/A is initiating minor compaction (all files) 2023-03-24 21:55:33,264 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:33,264 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/A in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,264 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46696 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:33,264 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:33,264 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:33,265 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/B is initiating minor compaction (all files) 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:33,264 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ef91bca5bbfc44ce98a78829abc5f0be, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=115.9 K 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:33,265 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/B in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:33,265 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a431861aebb4b1cb84454e0b42a45c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=45.6 K 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:33,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:33,265 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ef91bca5bbfc44ce98a78829abc5f0be, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927880 2023-03-24 21:55:33,266 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0e4b28d287b649959b4c2afa0429f1b8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679694928136 2023-03-24 21:55:33,266 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1a431861aebb4b1cb84454e0b42a45c9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927880 2023-03-24 21:55:33,267 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 108bcd3b81fc4d108c7dc2e9040278d4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679694930293 2023-03-24 21:55:33,268 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 85638c8f457244bb9fcd857f0a286010, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679694928136 2023-03-24 21:55:33,268 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7cb05120624641b1b4363772cd20ee75, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679694931954 2023-03-24 21:55:33,268 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 43adf993033b4e21ba4d8ca0ed5e342d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679694930294 2023-03-24 21:55:33,268 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting df8cfe79dcc94c4abdeaee695dc95962, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679694931957 2023-03-24 21:55:33,282 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#B#compaction#366 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:33,288 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240186468d154f44b5afca8dc7e902f9d8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240186468d154f44b5afca8dc7e902f9d8 2023-03-24 21:55:33,292 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#A#compaction#367 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:33,296 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/18b3cfc62dcb46298d5cb9c0eee360c2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/18b3cfc62dcb46298d5cb9c0eee360c2 2023-03-24 21:55:33,303 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1019, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/0474344a496d4de7aeac0da5332d6b43 2023-03-24 21:55:33,304 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/B of 3b6e2d0df6ef7b528783b27726388c56 into 18b3cfc62dcb46298d5cb9c0eee360c2(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:33,304 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,304 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/B, priority=12, startTime=1679694933262; duration=0sec 2023-03-24 21:55:33,304 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:33,304 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:33,306 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46662 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:33,306 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 3b6e2d0df6ef7b528783b27726388c56/C is initiating minor compaction (all files) 2023-03-24 21:55:33,306 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 3b6e2d0df6ef7b528783b27726388c56/C in TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,306 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ab01184fece94fb39e426f98250865f5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp, totalSize=45.6 K 2023-03-24 21:55:33,307 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ab01184fece94fb39e426f98250865f5, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=254, earliestPutTs=1679694927880 2023-03-24 21:55:33,307 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting fc13cd9c9fa84ab29c7e7d2a2ab02e1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1679694928136 2023-03-24 21:55:33,308 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting eb22d879f37145978bffe4dd7a7f35ef, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1679694930294 2023-03-24 21:55:33,308 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2df3c55cf7414a0a83f83566211c9b42, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1008, earliestPutTs=1679694931957 2023-03-24 21:55:33,337 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 3b6e2d0df6ef7b528783b27726388c56#C#compaction#369 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:33,354 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/fcecb90b352547f9819366e7dbc2a808 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fcecb90b352547f9819366e7dbc2a808 2023-03-24 21:55:33,359 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/C of 3b6e2d0df6ef7b528783b27726388c56 into fcecb90b352547f9819366e7dbc2a808(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:33,359 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,359 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/C, priority=12, startTime=1679694933263; duration=0sec 2023-03-24 21:55:33,359 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:33,748 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/94cb076f4a564caba0c1627f64d114bd 2023-03-24 21:55:33,770 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/e980b66ad84247859847d0a95155e765 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e980b66ad84247859847d0a95155e765 2023-03-24 21:55:33,777 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 3b6e2d0df6ef7b528783b27726388c56/A of 3b6e2d0df6ef7b528783b27726388c56 into e980b66ad84247859847d0a95155e765(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:33,777 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,777 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56., storeName=3b6e2d0df6ef7b528783b27726388c56/A, priority=12, startTime=1679694933262; duration=0sec 2023-03-24 21:55:33,777 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:33,806 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/5a5f0921ff5941ca9e154e48d49ccb44 2023-03-24 21:55:33,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/0474344a496d4de7aeac0da5332d6b43 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0474344a496d4de7aeac0da5332d6b43 2023-03-24 21:55:33,817 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0474344a496d4de7aeac0da5332d6b43, entries=150, sequenceid=1019, filesize=25.4 K 2023-03-24 21:55:33,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/94cb076f4a564caba0c1627f64d114bd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/94cb076f4a564caba0c1627f64d114bd 2023-03-24 21:55:33,822 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/94cb076f4a564caba0c1627f64d114bd, entries=150, sequenceid=1019, filesize=11.9 K 2023-03-24 21:55:33,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/5a5f0921ff5941ca9e154e48d49ccb44 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5a5f0921ff5941ca9e154e48d49ccb44 2023-03-24 21:55:33,827 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5a5f0921ff5941ca9e154e48d49ccb44, entries=150, sequenceid=1019, filesize=11.9 K 2023-03-24 21:55:33,827 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 3b6e2d0df6ef7b528783b27726388c56 in 563ms, sequenceid=1019, compaction requested=false 2023-03-24 21:55:33,827 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,828 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:33,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:33,828 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:33,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:33,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,837 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:33,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17958179[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:33,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:33,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,840 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:33,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:33,840 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:33,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:33,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:33,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:33,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:33,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:33,845 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:33,845 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:33,845 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:33,845 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:33,845 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:33,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:33,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:33,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:33,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:33,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,857 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:33,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:33,857 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:33,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,858 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,957 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:33,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:33,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:33,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:33,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:33,961 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:33,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:33,961 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:33,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,962 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:33,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:33,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:33,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,964 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,964 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:33,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:33,965 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:33,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:33,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:33,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:33,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:33,967 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:33,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:33,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:33,972 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:33,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:33,973 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:33,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:33,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:33,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:33,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:33,974 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:33,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a893742[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:33,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:33,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,979 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:33,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:33,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:33,979 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:33,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:33,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:33,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:33,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:33,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:33,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:33,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:33,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:33,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:33,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:33,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:33,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:33,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:34,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:34,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:34,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:34,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:34,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:34,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:34,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:34,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:34,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:34,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:34,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:34,006 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:34,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:34,006 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:34,006 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:34,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:34,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:34,007 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:34,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:34,107 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:34,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:34,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:34,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:34,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:34,111 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:34,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:34,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:34,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:34,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:34,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:34,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:34,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:34,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:34,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:34,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:34,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:34,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:34,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:34,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:34,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:34,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:34,127 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. started... 2023-03-24 21:55:34,127 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:34,128 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:34,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:34,191 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. as already flushing 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694934202, max timestamp=1679694934215Num uniques 859; 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694934202, max timestamp=1679694934215Num uniques 859; 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694934202, max timestamp=1679694934215Num uniques 859; 2023-03-24 21:55:34,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679694934233, max timestamp=1679694934265Num uniques 1068; 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679694934233, max timestamp=1679694934265Num uniques 1068; 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=318, cellSize=14558, totalHeapSize=33312, min timestamp=1679694934233, max timestamp=1679694934265Num uniques 1068; 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-03-24 21:55:34,269 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=14558, suffix heap size=139040,new segment heap size=33312 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=318 2023-03-24 21:55:34,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679694934294, max timestamp=1679694934318Num uniques 1177; 2023-03-24 21:55:34,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679694934294, max timestamp=1679694934318Num uniques 1177; 2023-03-24 21:55:34,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-03-24 21:55:34,321 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=14972, suffix heap size=150376,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=327 2023-03-24 21:55:34,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679694934309, max timestamp=1679694934318Num uniques 1177; 2023-03-24 21:55:34,324 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=53902, new segment data size=8102, suffix heap size=150376,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1177, new segment cells count=177 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1679694934349, max timestamp=1679694934359Num uniques 1186; 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=54306, new segment data size=8506, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1679694934349, max timestamp=1679694934359Num uniques 1186; 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8506, totalHeapSize=19584, min timestamp=1679694934349, max timestamp=1679694934359Num uniques 1036; 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=47436, new segment data size=8506, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-24 21:55:34,373 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=54306, new segment data size=8506, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-24 21:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694934381, max timestamp=1679694934404Num uniques 1045; 2023-03-24 21:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694934381, max timestamp=1679694934404Num uniques 1045; 2023-03-24 21:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694934381, max timestamp=1679694934404Num uniques 1045; 2023-03-24 21:55:34,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:34,407 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:34,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1679694934418, max timestamp=1679694934441Num uniques 1104; 2023-03-24 21:55:34,443 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=18500, suffix heap size=142784,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=404 2023-03-24 21:55:34,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1679694934418, max timestamp=1679694934441Num uniques 1104; 2023-03-24 21:55:34,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=404, cellSize=18500, totalHeapSize=42256, min timestamp=1679694934418, max timestamp=1679694934441Num uniques 1104; 2023-03-24 21:55:34,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=18500, suffix heap size=142784,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=404 2023-03-24 21:55:34,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=18500, suffix heap size=142784,new segment heap size=42256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=404 2023-03-24 21:55:34,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:34,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:37004 deadline: 1679694994466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:34,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694934463, max timestamp=1679694934472Num uniques 1263; 2023-03-24 21:55:34,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=9748, suffix heap size=159320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=213 2023-03-24 21:55:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694934463, max timestamp=1679694934472Num uniques 1263; 2023-03-24 21:55:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=9748, suffix heap size=159320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=213 2023-03-24 21:55:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1679694934463, max timestamp=1679694934472Num uniques 1263; 2023-03-24 21:55:34,475 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=57838, new segment data size=9748, suffix heap size=159320,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1263, new segment cells count=213 2023-03-24 21:55:34,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679694934495, max timestamp=1679694934514Num uniques 1072; 2023-03-24 21:55:34,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-03-24 21:55:34,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679694934495, max timestamp=1679694934514Num uniques 1072; 2023-03-24 21:55:34,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=322, cellSize=14742, totalHeapSize=33728, min timestamp=1679694934495, max timestamp=1679694934514Num uniques 1072; 2023-03-24 21:55:34,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-03-24 21:55:34,517 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=14742, suffix heap size=139456,new segment heap size=33728 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=322 2023-03-24 21:55:34,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694934536, max timestamp=1679694934553Num uniques 1181; 2023-03-24 21:55:34,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694934536, max timestamp=1679694934553Num uniques 1181; 2023-03-24 21:55:34,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-03-24 21:55:34,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694934536, max timestamp=1679694934553Num uniques 1181; 2023-03-24 21:55:34,555 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-03-24 21:55:34,556 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=54086, new segment data size=12866, suffix heap size=150792,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1181, new segment cells count=281 2023-03-24 21:55:34,575 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b70cf25d8703471085a65e6d8564bce2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b70cf25d8703471085a65e6d8564bce2 2023-03-24 21:55:34,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1679694934570, max timestamp=1679694934583Num uniques 1140; 2023-03-24 21:55:34,585 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1679694934570, max timestamp=1679694934583Num uniques 1140; 2023-03-24 21:55:34,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-24 21:55:34,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-24 21:55:34,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=390, cellSize=17860, totalHeapSize=40800, min timestamp=1679694934570, max timestamp=1679694934583Num uniques 1140; 2023-03-24 21:55:34,586 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=17860, suffix heap size=146528,new segment heap size=40800 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=390 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694934604, max timestamp=1679694934611Num uniques 1249; 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694934604, max timestamp=1679694934611Num uniques 1249; 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=299, cellSize=13694, totalHeapSize=31336, min timestamp=1679694934604, max timestamp=1679694934611Num uniques 1249; 2023-03-24 21:55:34,613 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=57204, new segment data size=13694, suffix heap size=157864,new segment heap size=31336 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1249, new segment cells count=299 2023-03-24 21:55:34,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694934633, max timestamp=1679694934641Num uniques 1158; 2023-03-24 21:55:34,643 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:34,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694934633, max timestamp=1679694934641Num uniques 1158; 2023-03-24 21:55:34,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679694934633, max timestamp=1679694934641Num uniques 1158; 2023-03-24 21:55:34,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:34,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=53030, new segment data size=11810, suffix heap size=148400,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1158, new segment cells count=258 2023-03-24 21:55:34,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679694934655, max timestamp=1679694934669Num uniques 1117; 2023-03-24 21:55:34,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679694934655, max timestamp=1679694934669Num uniques 1117; 2023-03-24 21:55:34,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-24 21:55:34,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-24 21:55:34,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679694934655, max timestamp=1679694934669Num uniques 1117; 2023-03-24 21:55:34,672 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=7642, suffix heap size=144136,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=167 2023-03-24 21:55:34,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679694934681, max timestamp=1679694934690Num uniques 1026; 2023-03-24 21:55:34,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-03-24 21:55:34,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679694934681, max timestamp=1679694934690Num uniques 1026; 2023-03-24 21:55:34,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679694934681, max timestamp=1679694934690Num uniques 1026; 2023-03-24 21:55:34,692 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-03-24 21:55:34,693 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=14926, suffix heap size=134672,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=326 2023-03-24 21:55:34,723 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694934697, max timestamp=1679694934722Num uniques 1185; 2023-03-24 21:55:34,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-24 21:55:34,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694934697, max timestamp=1679694934722Num uniques 1185; 2023-03-24 21:55:34,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679694934697, max timestamp=1679694934722Num uniques 1185; 2023-03-24 21:55:34,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-24 21:55:34,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-24 21:55:34,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679694934730, max timestamp=1679694934753Num uniques 1094; 2023-03-24 21:55:34,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 21:55:34,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8874, totalHeapSize=20416, min timestamp=1679694934730, max timestamp=1679694934753Num uniques 1094; 2023-03-24 21:55:34,755 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50094, new segment data size=8874, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 21:55:34,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8882, totalHeapSize=20416, min timestamp=1679694934730, max timestamp=1679694934753Num uniques 1094; 2023-03-24 21:55:34,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50102, new segment data size=8882, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-03-24 21:55:34,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679694934774, max timestamp=1679694934781Num uniques 1053; 2023-03-24 21:55:34,782 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 21:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679694934774, max timestamp=1679694934781Num uniques 1053; 2023-03-24 21:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=153, cellSize=7005, totalHeapSize=16152, min timestamp=1679694934774, max timestamp=1679694934781Num uniques 1053; 2023-03-24 21:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 21:55:34,783 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=48225, new segment data size=7005, suffix heap size=137480,new segment heap size=16152 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1053, new segment cells count=153 2023-03-24 21:55:34,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694934806, max timestamp=1679694934812Num uniques 1012; 2023-03-24 21:55:34,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694934806, max timestamp=1679694934812Num uniques 1012; 2023-03-24 21:55:34,814 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 21:55:34,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 21:55:34,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1679694934806, max timestamp=1679694934812Num uniques 1012; 2023-03-24 21:55:34,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46342, new segment data size=7412, suffix heap size=133216,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1012, new segment cells count=162 2023-03-24 21:55:34,865 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694934815, max timestamp=1679694934864Num uniques 1021; 2023-03-24 21:55:34,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:34,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694934815, max timestamp=1679694934864Num uniques 1021; 2023-03-24 21:55:34,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679694934815, max timestamp=1679694934864Num uniques 1021; 2023-03-24 21:55:34,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:34,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-03-24 21:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1679694934880, max timestamp=1679694934895Num uniques 1080; 2023-03-24 21:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1679694934880, max timestamp=1679694934895Num uniques 1080; 2023-03-24 21:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-03-24 21:55:34,897 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-03-24 21:55:34,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1679694934880, max timestamp=1679694934895Num uniques 1080; 2023-03-24 21:55:34,898 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-03-24 21:55:34,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694934909, max timestamp=1679694934923Num uniques 1139; 2023-03-24 21:55:34,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-03-24 21:55:34,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694934909, max timestamp=1679694934923Num uniques 1139; 2023-03-24 21:55:34,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-03-24 21:55:34,925 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1679694934909, max timestamp=1679694934923Num uniques 1139; 2023-03-24 21:55:34,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679694934940, max timestamp=1679694934948Num uniques 1098; 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679694934940, max timestamp=1679694934948Num uniques 1098; 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=248, cellSize=11358, totalHeapSize=26032, min timestamp=1679694934940, max timestamp=1679694934948Num uniques 1098; 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-03-24 21:55:34,951 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50288, new segment data size=11358, suffix heap size=142160,new segment heap size=26032 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1098, new segment cells count=248 2023-03-24 21:55:34,973 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694934965, max timestamp=1679694934972Num uniques 1107; 2023-03-24 21:55:34,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-24 21:55:34,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694934965, max timestamp=1679694934972Num uniques 1107; 2023-03-24 21:55:34,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-24 21:55:34,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=257, cellSize=11765, totalHeapSize=26968, min timestamp=1679694934965, max timestamp=1679694934972Num uniques 1107; 2023-03-24 21:55:34,974 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=50695, new segment data size=11765, suffix heap size=143096,new segment heap size=26968 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1107, new segment cells count=257 2023-03-24 21:55:34,981 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1027, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/35e7d160ce7f492380109d7f3dbcd7f4 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679694934993, max timestamp=1679694935027Num uniques 1116; 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=12176, suffix heap size=144032,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=266 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679694934993, max timestamp=1679694935027Num uniques 1116; 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=266, cellSize=12176, totalHeapSize=27904, min timestamp=1679694934993, max timestamp=1679694935027Num uniques 1116; 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=12176, suffix heap size=144032,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=266 2023-03-24 21:55:35,030 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=51106, new segment data size=12176, suffix heap size=144032,new segment heap size=27904 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1116, new segment cells count=266 2023-03-24 21:55:35,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694935056, max timestamp=1679694935063Num uniques 1125; 2023-03-24 21:55:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-24 21:55:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694935056, max timestamp=1679694935063Num uniques 1125; 2023-03-24 21:55:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-24 21:55:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=225, cellSize=10300, totalHeapSize=23640, min timestamp=1679694935056, max timestamp=1679694935063Num uniques 1125; 2023-03-24 21:55:35,066 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=51520, new segment data size=10300, suffix heap size=144968,new segment heap size=23640 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1125, new segment cells count=225 2023-03-24 21:55:35,115 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694935082, max timestamp=1679694935113Num uniques 1084; 2023-03-24 21:55:35,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:35,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694935082, max timestamp=1679694935113Num uniques 1084; 2023-03-24 21:55:35,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:35,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=234, cellSize=10714, totalHeapSize=24576, min timestamp=1679694935082, max timestamp=1679694935113Num uniques 1084; 2023-03-24 21:55:35,116 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=49644, new segment data size=10714, suffix heap size=140704,new segment heap size=24576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1084, new segment cells count=234 2023-03-24 21:55:35,143 DEBUG [Thread-2370] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2879fb80 to 127.0.0.1:57724 2023-03-24 21:55:35,143 DEBUG [Thread-2389] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34018a0c to 127.0.0.1:57724 2023-03-24 21:55:35,143 DEBUG [Thread-2370] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,143 DEBUG [Thread-2389] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,144 DEBUG [Thread-2372] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ded5508 to 127.0.0.1:57724 2023-03-24 21:55:35,144 DEBUG [Thread-2372] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=343, cellSize=15708, totalHeapSize=35912, min timestamp=1679694935130, max timestamp=1679694935143Num uniques 1093; 2023-03-24 21:55:35,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=15708, suffix heap size=141640,new segment heap size=35912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=343 2023-03-24 21:55:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=343, cellSize=15708, totalHeapSize=35912, min timestamp=1679694935130, max timestamp=1679694935143Num uniques 1093; 2023-03-24 21:55:35,146 DEBUG [Thread-2383] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01035478 to 127.0.0.1:57724 2023-03-24 21:55:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=15708, suffix heap size=141640,new segment heap size=35912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=343 2023-03-24 21:55:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=343, cellSize=15708, totalHeapSize=35912, min timestamp=1679694935130, max timestamp=1679694935143Num uniques 1093; 2023-03-24 21:55:35,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=50058, new segment data size=15708, suffix heap size=141640,new segment heap size=35912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1093, new segment cells count=343 2023-03-24 21:55:35,146 DEBUG [Thread-2383] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,147 DEBUG [Thread-2378] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x154073b2 to 127.0.0.1:57724 2023-03-24 21:55:35,147 DEBUG [Thread-2378] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,149 DEBUG [Thread-2376] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3be39f87 to 127.0.0.1:57724 2023-03-24 21:55:35,149 DEBUG [Thread-2376] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,149 DEBUG [Thread-2381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x636cfa8a to 127.0.0.1:57724 2023-03-24 21:55:35,149 DEBUG [Thread-2381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,154 DEBUG [Thread-2385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72f4141b to 127.0.0.1:57724 2023-03-24 21:55:35,154 DEBUG [Thread-2385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,154 DEBUG [Thread-2387] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01ad7293 to 127.0.0.1:57724 2023-03-24 21:55:35,154 DEBUG [Thread-2387] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,390 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1027 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/dec85f3e66e647ddad5eeecec38f5469 2023-03-24 21:55:35,798 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=1027 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/0988f87b26234adcb16f2fccbf3eb8a1 2023-03-24 21:55:35,801 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/35e7d160ce7f492380109d7f3dbcd7f4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/35e7d160ce7f492380109d7f3dbcd7f4 2023-03-24 21:55:35,804 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/35e7d160ce7f492380109d7f3dbcd7f4, entries=50, sequenceid=1027, filesize=11.7 K 2023-03-24 21:55:35,804 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/dec85f3e66e647ddad5eeecec38f5469 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/dec85f3e66e647ddad5eeecec38f5469 2023-03-24 21:55:35,806 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/dec85f3e66e647ddad5eeecec38f5469, entries=50, sequenceid=1027, filesize=7.1 K 2023-03-24 21:55:35,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/0988f87b26234adcb16f2fccbf3eb8a1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/0988f87b26234adcb16f2fccbf3eb8a1 2023-03-24 21:55:35,810 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/0988f87b26234adcb16f2fccbf3eb8a1, entries=50, sequenceid=1027, filesize=7.1 K 2023-03-24 21:55:35,811 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=60.38 KB/61830 for 3b6e2d0df6ef7b528783b27726388c56 in 1684ms, sequenceid=1027, compaction requested=true 2023-03-24 21:55:35,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:35,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:35,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:35,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:35,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:35,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:35,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,813 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:35,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:35,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:35,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17167708[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:35,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:35,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,818 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,818 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:35,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,818 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:35,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:35,818 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:35,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:35,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:35,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:35,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:35,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:35,822 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:35,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:35,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:35,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:35,822 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:35,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,823 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:35,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:35,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:35,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:35,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:35,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:35,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:35,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:35,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:35,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:35,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:35,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:35,827 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:35,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:35,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:35,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:35,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:35,828 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,828 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,828 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:35,828 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:35,828 DEBUG [Thread-2380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:35,828 DEBUG [Thread-2380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:35,928 DEBUG [Thread-2380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:35,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:35,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:35,976 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 3b6e2d0df6ef7b528783b27726388c56 3/3 column families, dataSize=67.09 KB heapSize=168.52 KB 2023-03-24 21:55:35,976 DEBUG [Thread-2374] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x484332e6 to 127.0.0.1:57724 2023-03-24 21:55:35,976 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=A 2023-03-24 21:55:35,977 DEBUG [Thread-2374] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:55:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=B 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 334 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 317 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 338 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 377 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1808 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5424 rows 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1800 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1792 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5376 rows 2023-03-24 21:55:35,977 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-03-24 21:55:35,978 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-03-24 21:55:35,977 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:35,978 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:55:35,978 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 3b6e2d0df6ef7b528783b27726388c56, store=C 2023-03-24 21:55:35,978 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x016ba88c to 127.0.0.1:57724 2023-03-24 21:55:35,978 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:35,978 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:55:35,979 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:55:35,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:55:35,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:35,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:35,985 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694935984"}]},"ts":"1679694935984"} 2023-03-24 21:55:35,988 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:55:35,989 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:55:35,990 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, UNASSIGN}] 2023-03-24 21:55:35,995 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, UNASSIGN 2023-03-24 21:55:35,996 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:35,996 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694935996"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694935996"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694935996"}]},"ts":"1679694935996"} 2023-03-24 21:55:35,997 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:36,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:36,149 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:36,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 3b6e2d0df6ef7b528783b27726388c56, disabling compactions & flushes 2023-03-24 21:55:36,149 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:36,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:36,401 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324936e292d2c814c399181af314685c04b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324936e292d2c814c399181af314685c04b 2023-03-24 21:55:36,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:36,806 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1497, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/593c496683be427b88bbc0ebcb7de63e 2023-03-24 21:55:37,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:37,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1497 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/26df773852af4c9bae3b6fd218de7237 2023-03-24 21:55:37,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1497 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/531f44c1c7274123acc7414745c007a0 2023-03-24 21:55:37,628 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/A/593c496683be427b88bbc0ebcb7de63e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/593c496683be427b88bbc0ebcb7de63e 2023-03-24 21:55:37,631 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/593c496683be427b88bbc0ebcb7de63e, entries=150, sequenceid=1497, filesize=25.4 K 2023-03-24 21:55:37,632 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/B/26df773852af4c9bae3b6fd218de7237 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/26df773852af4c9bae3b6fd218de7237 2023-03-24 21:55:37,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/26df773852af4c9bae3b6fd218de7237, entries=150, sequenceid=1497, filesize=11.9 K 2023-03-24 21:55:37,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/.tmp/C/531f44c1c7274123acc7414745c007a0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/531f44c1c7274123acc7414745c007a0 2023-03-24 21:55:37,638 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/531f44c1c7274123acc7414745c007a0, entries=150, sequenceid=1497, filesize=11.9 K 2023-03-24 21:55:37,639 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~168.47 KB/172512, currentSize=0 B/0 for 3b6e2d0df6ef7b528783b27726388c56 in 1663ms, sequenceid=1497, compaction requested=true 2023-03-24 21:55:37,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:37,639 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:37,639 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. because compaction request was cancelled 2023-03-24 21:55:37,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:37,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:37,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. after waiting 0 ms 2023-03-24 21:55:37,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:37,639 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:37,639 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. because compaction request was cancelled 2023-03-24 21:55:37,639 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. because compaction request was cancelled 2023-03-24 21:55:37,639 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:37,660 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b0df0325236e4052aa77547baa73b66e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/8945ec32c21d4ace80c087ad68568d78, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/f39765cc3ba340e0a00e863e172602ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2ec22c0a1c3d47e6b21b16742e71b83c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ef91bca5bbfc44ce98a78829abc5f0be, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75] to archive 2023-03-24 21:55:37,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:37,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2e6427ecfbe84cffa90f306b94d21301 2023-03-24 21:55:37,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ebdc2074d42441a79d5d018fb6aac675 2023-03-24 21:55:37,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/66a2c5d65de34beeb2706bb863a5393b 2023-03-24 21:55:37,665 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b0df0325236e4052aa77547baa73b66e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b0df0325236e4052aa77547baa73b66e 2023-03-24 21:55:37,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/cc67e8c75f234b54856888737e933234 2023-03-24 21:55:37,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/c10510f4d88e4dc09caaa96a8b022505 2023-03-24 21:55:37,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/8945ec32c21d4ace80c087ad68568d78 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/8945ec32c21d4ace80c087ad68568d78 2023-03-24 21:55:37,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e9f4b53dbb914c06a7099f168539fd4e 2023-03-24 21:55:37,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/3b70ee8f1d1745e2aa1c168355f39a8e 2023-03-24 21:55:37,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/f39765cc3ba340e0a00e863e172602ec to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/f39765cc3ba340e0a00e863e172602ec 2023-03-24 21:55:37,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/af081023a9bb4822a54f136902ce21ee 2023-03-24 21:55:37,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/bc78da9f94a1462c9707e293dc049940 2023-03-24 21:55:37,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/65e8fa87c6bd4f05913b11fb97efeb45 2023-03-24 21:55:37,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2ec22c0a1c3d47e6b21b16742e71b83c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/2ec22c0a1c3d47e6b21b16742e71b83c 2023-03-24 21:55:37,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/b2d960675d0145f8b43ee406b5cfdd7e 2023-03-24 21:55:37,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/50d9fbb14f5f4dd1a37070f62fed15c9 2023-03-24 21:55:37,677 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ef91bca5bbfc44ce98a78829abc5f0be to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/ef91bca5bbfc44ce98a78829abc5f0be 2023-03-24 21:55:37,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0e4b28d287b649959b4c2afa0429f1b8 2023-03-24 21:55:37,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/108bcd3b81fc4d108c7dc2e9040278d4 2023-03-24 21:55:37,680 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/7cb05120624641b1b4363772cd20ee75 2023-03-24 21:55:37,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8af3d20d45cb495281f4a77e4f1a90a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/3a0d88b4557e450bab8b12c4d76075a1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/adf3ed932927414e8dfd80c0d76aee1d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1e692b2e1f554f5e9031dd4c4e9d2356, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a431861aebb4b1cb84454e0b42a45c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962] to archive 2023-03-24 21:55:37,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:37,684 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8ac7eaf333af4fdb8172aaf6c457446a 2023-03-24 21:55:37,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9af7332c75ca4b8384fe8e3ff9bb447a 2023-03-24 21:55:37,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8af3d20d45cb495281f4a77e4f1a90a5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8af3d20d45cb495281f4a77e4f1a90a5 2023-03-24 21:55:37,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/8e2bd0ec32e94b5da72b5e57e70b6ea2 2023-03-24 21:55:37,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/60ff7b3cba9f48e6bbe6312c47b7b002 2023-03-24 21:55:37,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/3a0d88b4557e450bab8b12c4d76075a1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/3a0d88b4557e450bab8b12c4d76075a1 2023-03-24 21:55:37,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a64e628d7744535a4f1f889d7bc6021 2023-03-24 21:55:37,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/5f76b6c5982a451d880a52c8d52de902 2023-03-24 21:55:37,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/adf3ed932927414e8dfd80c0d76aee1d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/adf3ed932927414e8dfd80c0d76aee1d 2023-03-24 21:55:37,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/bac625162af44b92ab5ba93a075f07e2 2023-03-24 21:55:37,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/99d4bb5a65d74cddaa1e323854e7fffd 2023-03-24 21:55:37,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a65ef73393b04e5590ab5529d28117c9 2023-03-24 21:55:37,693 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1e692b2e1f554f5e9031dd4c4e9d2356 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1e692b2e1f554f5e9031dd4c4e9d2356 2023-03-24 21:55:37,694 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/a6dc51799de2436e98bd5492202e50a0 2023-03-24 21:55:37,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/ed2cef67e266403d97c9dcb7b41ecb50 2023-03-24 21:55:37,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a431861aebb4b1cb84454e0b42a45c9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/1a431861aebb4b1cb84454e0b42a45c9 2023-03-24 21:55:37,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/9637fcf51e084751a61b23d07b182559 2023-03-24 21:55:37,697 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/85638c8f457244bb9fcd857f0a286010 2023-03-24 21:55:37,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/43adf993033b4e21ba4d8ca0ed5e342d 2023-03-24 21:55:37,698 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/df8cfe79dcc94c4abdeaee695dc95962 2023-03-24 21:55:37,700 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/381fc0a1851040619756e1c1edd76cd0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5ba8f1bcf5f2492587a8743f58f44252, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/df9257605f9f4c4198813187ee1bf437, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ab01184fece94fb39e426f98250865f5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42] to archive 2023-03-24 21:55:37,701 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:55:37,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/8bb634433c1b423e90c9906eaf40c810 2023-03-24 21:55:37,703 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/b9b1b61d5ae94d5ea5235e8b7a643b1d 2023-03-24 21:55:37,704 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2f53ffbc665c4c2bbb1abd5b45d96043 2023-03-24 21:55:37,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/381fc0a1851040619756e1c1edd76cd0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/381fc0a1851040619756e1c1edd76cd0 2023-03-24 21:55:37,705 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ff1b6dc6332e4cd88d970e1f52466053 2023-03-24 21:55:37,706 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fefab3a6b1cc4af9a47a10bc336b025f 2023-03-24 21:55:37,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/6cf6b2593e564195ba8536731fd8ce6e 2023-03-24 21:55:37,707 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5ba8f1bcf5f2492587a8743f58f44252 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5ba8f1bcf5f2492587a8743f58f44252 2023-03-24 21:55:37,708 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/51a101f77a5443aea1359be0fd138462 2023-03-24 21:55:37,709 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/168a3b2eb40f4002b74ddc9a4102ad5c 2023-03-24 21:55:37,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/d796e15018fa4fad910344c36423bf95 2023-03-24 21:55:37,710 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/df9257605f9f4c4198813187ee1bf437 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/df9257605f9f4c4198813187ee1bf437 2023-03-24 21:55:37,711 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/7061e9405d3a4b519de93cae19551ea8 2023-03-24 21:55:37,712 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/53bc5ff2c42749f9b4f7e863bb3099d4 2023-03-24 21:55:37,713 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ab01184fece94fb39e426f98250865f5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/ab01184fece94fb39e426f98250865f5 2023-03-24 21:55:37,714 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/dc2b1314534e4bd4968bbbbfc437d861 2023-03-24 21:55:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fc13cd9c9fa84ab29c7e7d2a2ab02e1e 2023-03-24 21:55:37,715 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/eb22d879f37145978bffe4dd7a7f35ef 2023-03-24 21:55:37,716 DEBUG [StoreCloser-TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/2df3c55cf7414a0a83f83566211c9b42 2023-03-24 21:55:37,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits/1500.seqid, newMaxSeqId=1500, maxSeqId=4 2023-03-24 21:55:37,722 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56. 2023-03-24 21:55:37,722 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 3b6e2d0df6ef7b528783b27726388c56: 2023-03-24 21:55:37,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:37,723 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=3b6e2d0df6ef7b528783b27726388c56, regionState=CLOSED 2023-03-24 21:55:37,723 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694937723"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694937723"}]},"ts":"1679694937723"} 2023-03-24 21:55:37,725 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-24 21:55:37,725 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 3b6e2d0df6ef7b528783b27726388c56, server=jenkins-hbase9.apache.org,45673,1679694819052 in 1.7270 sec 2023-03-24 21:55:37,726 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-24 21:55:37,726 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=3b6e2d0df6ef7b528783b27726388c56, UNASSIGN in 1.7350 sec 2023-03-24 21:55:37,727 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694937727"}]},"ts":"1679694937727"} 2023-03-24 21:55:37,728 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:55:37,729 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:55:37,730 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7490 sec 2023-03-24 21:55:38,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-24 21:55:38,089 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-24 21:55:38,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:55:38,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,092 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 21:55:38,093 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,096 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:38,098 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits] 2023-03-24 21:55:38,100 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0474344a496d4de7aeac0da5332d6b43 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/0474344a496d4de7aeac0da5332d6b43 2023-03-24 21:55:38,101 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/35e7d160ce7f492380109d7f3dbcd7f4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/35e7d160ce7f492380109d7f3dbcd7f4 2023-03-24 21:55:38,102 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/593c496683be427b88bbc0ebcb7de63e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/593c496683be427b88bbc0ebcb7de63e 2023-03-24 21:55:38,103 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e980b66ad84247859847d0a95155e765 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/A/e980b66ad84247859847d0a95155e765 2023-03-24 21:55:38,105 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/18b3cfc62dcb46298d5cb9c0eee360c2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/18b3cfc62dcb46298d5cb9c0eee360c2 2023-03-24 21:55:38,106 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/26df773852af4c9bae3b6fd218de7237 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/26df773852af4c9bae3b6fd218de7237 2023-03-24 21:55:38,107 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/94cb076f4a564caba0c1627f64d114bd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/94cb076f4a564caba0c1627f64d114bd 2023-03-24 21:55:38,108 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/dec85f3e66e647ddad5eeecec38f5469 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/B/dec85f3e66e647ddad5eeecec38f5469 2023-03-24 21:55:38,109 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/0988f87b26234adcb16f2fccbf3eb8a1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/0988f87b26234adcb16f2fccbf3eb8a1 2023-03-24 21:55:38,110 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/531f44c1c7274123acc7414745c007a0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/531f44c1c7274123acc7414745c007a0 2023-03-24 21:55:38,111 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5a5f0921ff5941ca9e154e48d49ccb44 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/5a5f0921ff5941ca9e154e48d49ccb44 2023-03-24 21:55:38,112 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fcecb90b352547f9819366e7dbc2a808 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/C/fcecb90b352547f9819366e7dbc2a808 2023-03-24 21:55:38,114 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits/1500.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56/recovered.edits/1500.seqid 2023-03-24 21:55:38,115 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/3b6e2d0df6ef7b528783b27726388c56 2023-03-24 21:55:38,115 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:55:38,115 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:55:38,116 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 21:55:38,118 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240186468d154f44b5afca8dc7e902f9d8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240186468d154f44b5afca8dc7e902f9d8 2023-03-24 21:55:38,119 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240b9333fba47c4dc39ce18d0b31d0ba5b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240b9333fba47c4dc39ce18d0b31d0ba5b 2023-03-24 21:55:38,119 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032411be30e839ee473fa362a2be47b54715 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032411be30e839ee473fa362a2be47b54715 2023-03-24 21:55:38,120 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428ce9e48b81e4af9b4f6989cee001fc6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032428ce9e48b81e4af9b4f6989cee001fc6 2023-03-24 21:55:38,121 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032478355d8e17a24362b94c71eaa99b4753 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032478355d8e17a24362b94c71eaa99b4753 2023-03-24 21:55:38,122 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248916492acdaa408a90eba96b2ee15165 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248916492acdaa408a90eba96b2ee15165 2023-03-24 21:55:38,122 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248dcaba8859f844d78adb6f8abc7d1f57 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303248dcaba8859f844d78adb6f8abc7d1f57 2023-03-24 21:55:38,123 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324936e292d2c814c399181af314685c04b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324936e292d2c814c399181af314685c04b 2023-03-24 21:55:38,124 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e5ff70f4a93483aa16071d0c5e4ccd7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e5ff70f4a93483aa16071d0c5e4ccd7 2023-03-24 21:55:38,125 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f1d9821809e4653a010fb01c7eb4ac5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249f1d9821809e4653a010fb01c7eb4ac5 2023-03-24 21:55:38,126 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b43581455a3d423183f59f4dd2a09fe0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b43581455a3d423183f59f4dd2a09fe0 2023-03-24 21:55:38,126 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b70cf25d8703471085a65e6d8564bce2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b70cf25d8703471085a65e6d8564bce2 2023-03-24 21:55:38,127 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c8ec1dd526754fb183c4f7f18645a852 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c8ec1dd526754fb183c4f7f18645a852 2023-03-24 21:55:38,128 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cb07a9f96369456cafb21268793838dc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cb07a9f96369456cafb21268793838dc 2023-03-24 21:55:38,129 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d65ee39d02de4154bc1cfbbd3d70502a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324d65ee39d02de4154bc1cfbbd3d70502a 2023-03-24 21:55:38,130 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9bdd51da25841daaedeed6e309d650f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e9bdd51da25841daaedeed6e309d650f 2023-03-24 21:55:38,131 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f3e5a013ddda44f0941541922a5c6420 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f3e5a013ddda44f0941541922a5c6420 2023-03-24 21:55:38,132 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f9f97f4fc9f14fbfbb6e3e200034de44 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f9f97f4fc9f14fbfbb6e3e200034de44 2023-03-24 21:55:38,132 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:55:38,134 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,135 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:55:38,137 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:55:38,138 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,138 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:55:38,138 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694938138"}]},"ts":"9223372036854775807"} 2023-03-24 21:55:38,139 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:55:38,139 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 3b6e2d0df6ef7b528783b27726388c56, NAME => 'TestAcidGuarantees,,1679694912042.3b6e2d0df6ef7b528783b27726388c56.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:55:38,139 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:55:38,139 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694938139"}]},"ts":"9223372036854775807"} 2023-03-24 21:55:38,143 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:55:38,144 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 54 msec 2023-03-24 21:55:38,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-24 21:55:38,194 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-24 21:55:38,205 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 226) - Thread LEAK? -, OpenFileDescriptor=544 (was 531) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 318) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=7854 (was 7930) 2023-03-24 21:55:38,216 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=169, AvailableMemoryMB=7853 2023-03-24 21:55:38,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:55:38,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:55:38,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:55:38,220 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:55:38,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-24 21:55:38,220 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:55:38,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:38,221 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:38,222 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af empty. 2023-03-24 21:55:38,222 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:38,222 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:55:38,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:38,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:38,629 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:55:38,630 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2a79b669e50783c6f283e5def38723af, NAME => 'TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:55:38,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2a79b669e50783c6f283e5def38723af, disabling compactions & flushes 2023-03-24 21:55:39,036 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. after waiting 0 ms 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,036 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,036 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:39,038 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:55:39,039 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694939039"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694939039"}]},"ts":"1679694939039"} 2023-03-24 21:55:39,040 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:55:39,040 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:55:39,041 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694939040"}]},"ts":"1679694939040"} 2023-03-24 21:55:39,041 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:55:39,045 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, ASSIGN}] 2023-03-24 21:55:39,046 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, ASSIGN 2023-03-24 21:55:39,047 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:55:39,198 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2a79b669e50783c6f283e5def38723af, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:39,198 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694939198"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694939198"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694939198"}]},"ts":"1679694939198"} 2023-03-24 21:55:39,199 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:55:39,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:39,354 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2a79b669e50783c6f283e5def38723af, NAME => 'TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:55:39,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:55:39,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,354 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,355 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,356 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A 2023-03-24 21:55:39,356 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A 2023-03-24 21:55:39,357 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:39,357 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a79b669e50783c6f283e5def38723af columnFamilyName A 2023-03-24 21:55:39,357 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(310): Store=2a79b669e50783c6f283e5def38723af/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:39,357 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,358 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B 2023-03-24 21:55:39,358 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B 2023-03-24 21:55:39,358 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:39,359 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a79b669e50783c6f283e5def38723af columnFamilyName B 2023-03-24 21:55:39,359 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(310): Store=2a79b669e50783c6f283e5def38723af/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:39,359 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,360 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C 2023-03-24 21:55:39,360 DEBUG [StoreOpener-2a79b669e50783c6f283e5def38723af-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C 2023-03-24 21:55:39,360 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:55:39,360 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2a79b669e50783c6f283e5def38723af columnFamilyName C 2023-03-24 21:55:39,361 INFO [StoreOpener-2a79b669e50783c6f283e5def38723af-1] regionserver.HStore(310): Store=2a79b669e50783c6f283e5def38723af/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:55:39,361 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,361 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,362 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,363 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:55:39,364 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:39,365 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:55:39,365 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 2a79b669e50783c6f283e5def38723af; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69936547, jitterRate=0.04213576018810272}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:55:39,366 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:39,366 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., pid=49, masterSystemTime=1679694939351 2023-03-24 21:55:39,367 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,367 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:39,368 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=2a79b669e50783c6f283e5def38723af, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:39,368 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694939367"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694939367"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694939367"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694939367"}]},"ts":"1679694939367"} 2023-03-24 21:55:39,369 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-24 21:55:39,370 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 in 169 msec 2023-03-24 21:55:39,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-24 21:55:39,370 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, ASSIGN in 325 msec 2023-03-24 21:55:39,371 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:55:39,371 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694939371"}]},"ts":"1679694939371"} 2023-03-24 21:55:39,372 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:55:39,373 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:55:39,374 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-03-24 21:55:40,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-24 21:55:40,325 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-24 21:55:40,325 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x65df9e98 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,331 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@20c4cfb4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,332 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,334 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:37712, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,335 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:55:40,337 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:50204, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:55:40,338 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x110b63af to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,341 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8390448, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,342 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5bd64911 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,345 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@110a5fdf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,345 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x6973f12f to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,349 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@84b3d1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,349 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1aac3cbf to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,353 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c204c4c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,353 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1575f790 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,357 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e476845, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,357 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x08344d0d to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,360 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5de4190e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,361 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x08ef5a5a to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,364 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2299226f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,364 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x1c2afc3b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,373 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6e65586e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,373 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x19cafcd7 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,378 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52ca10f1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,378 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x37210381 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:55:40,382 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@79436df6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:55:40,386 DEBUG [hconnection-0x2ee7eca3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,387 DEBUG [hconnection-0x703c3f32-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:40,388 DEBUG [hconnection-0x4463bff4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,388 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48156, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,388 DEBUG [hconnection-0x5fec7056-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,391 DEBUG [hconnection-0x4d67feb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,392 DEBUG [hconnection-0x2671f361-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,392 DEBUG [hconnection-0x518bded4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,392 DEBUG [hconnection-0x49c65ca6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,392 DEBUG [hconnection-0x368e1f95-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,393 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48178, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,393 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48210, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,398 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48142, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,398 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48176, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,399 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48196, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,398 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48170, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,399 DEBUG [hconnection-0x20ed664f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:55:40,399 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,400 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48226, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:40,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:40,405 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:40,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:40,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:40,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:40,405 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:48234, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:55:40,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:40,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:40,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,409 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:40,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:40,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:40,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:40,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:40,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:40,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:40,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:40,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:40,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:40,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:40,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:40,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:40,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:40,412 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:40,412 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:40,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:40,415 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:40,421 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/95dea600d70e437dbdcd2f57d7508e51 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679694940421, max timestamp=1679694940433Num uniques 859; 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679694940421, max timestamp=1679694940433Num uniques 859; 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1679694940421, max timestamp=1679694940433Num uniques 859; 2023-03-24 21:55:40,435 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-03-24 21:55:40,447 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695000444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695000444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695000445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695000447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695000447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695000548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695000550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695000551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695000551, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,553 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695000553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695000756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695000756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695000756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695000756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:40,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695000756, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:40,843 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/dfad0585b41b4f318423845439d89de7 2023-03-24 21:55:41,044 DEBUG [regionserver/jenkins-hbase9:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-24 21:55:41,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695001062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695001062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695001063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695001063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695001064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,253 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e0fa8904fa4e4c4aa65117d2dad5cbcd 2023-03-24 21:55:41,257 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/95dea600d70e437dbdcd2f57d7508e51 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51 2023-03-24 21:55:41,261 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51, entries=150, sequenceid=10, filesize=11.6 K 2023-03-24 21:55:41,261 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/dfad0585b41b4f318423845439d89de7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7 2023-03-24 21:55:41,264 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7, entries=150, sequenceid=10, filesize=11.6 K 2023-03-24 21:55:41,265 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e0fa8904fa4e4c4aa65117d2dad5cbcd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd 2023-03-24 21:55:41,268 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd, entries=150, sequenceid=10, filesize=11.6 K 2023-03-24 21:55:41,269 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 2a79b669e50783c6f283e5def38723af in 857ms, sequenceid=10, compaction requested=false 2023-03-24 21:55:41,269 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:41,269 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,269 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:41,269 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:41,269 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,269 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:41,269 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:41,270 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,271 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,271 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:41,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:41,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726127a8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:41,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:41,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:41,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:41,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:41,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:41,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:41,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,279 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,279 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:41,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:41,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:41,281 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:41,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:41,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:41,281 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:41,282 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:41,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:41,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,288 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:41,288 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:41,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:41,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,288 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,388 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:41,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:41,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:41,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:41,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:41,392 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:41,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:41,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:41,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:41,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:41,396 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:41,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:41,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:41,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:41,396 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:41,396 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,396 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:41,396 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=147.60 KB heapSize=396.70 KB 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:41,397 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,406 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/37d94f94ed9b47739682e1f7085fe16a 2023-03-24 21:55:41,433 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/673bd2f68d0944aebdb3a523189728eb 2023-03-24 21:55:41,455 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=43 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ad68c47ac8d74432b3d65dcbaceea407 2023-03-24 21:55:41,465 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/37d94f94ed9b47739682e1f7085fe16a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a 2023-03-24 21:55:41,471 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a, entries=150, sequenceid=43, filesize=11.6 K 2023-03-24 21:55:41,473 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/673bd2f68d0944aebdb3a523189728eb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb 2023-03-24 21:55:41,483 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb, entries=150, sequenceid=43, filesize=11.6 K 2023-03-24 21:55:41,484 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ad68c47ac8d74432b3d65dcbaceea407 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407 2023-03-24 21:55:41,489 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407, entries=150, sequenceid=43, filesize=11.6 K 2023-03-24 21:55:41,489 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~396.66 KB/406176, currentSize=0 B/0 for 2a79b669e50783c6f283e5def38723af in 93ms, sequenceid=43, compaction requested=false 2023-03-24 21:55:41,489 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:41,490 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:41,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:41,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:41,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:41,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,492 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:41,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4944eed4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:41,495 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:41,495 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,496 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:41,496 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:41,497 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:41,497 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:41,498 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,498 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:41,502 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:41,502 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:41,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:41,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:41,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:41,502 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:41,504 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,504 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,504 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,505 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:41,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,509 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:41,509 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:41,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:41,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:41,574 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:41,575 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:41,595 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/195c489c149f485792a3e9c1223f8bfe 2023-03-24 21:55:41,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695001599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694941596, max timestamp=1679694941604Num uniques 859; 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694941596, max timestamp=1679694941604Num uniques 859; 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694941596, max timestamp=1679694941604Num uniques 859; 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:41,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:41,609 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:41,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:41,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:41,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/010b16c47fad42b698ae9620478b3750 2023-03-24 21:55:41,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:41,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:41,650 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:41,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:41,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:41,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:41,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695001639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,653 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695001639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,654 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:41,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:41,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:41,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:41,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,654 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:41,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695001652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,666 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695001653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,666 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:41,666 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:41,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:41,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:41,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,667 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:41,667 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:41,667 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,667 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:41,667 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:41,669 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,669 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:41,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af4f399[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:41,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:41,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,673 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:41,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:41,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:41,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,674 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:41,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:41,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:41,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:41,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:41,678 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:41,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:41,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:41,678 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:41,678 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:41,679 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,680 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,682 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:41,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:41,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:41,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,689 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:41,689 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=54 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/34663e532a0048528e6c15f7d43ce27d 2023-03-24 21:55:41,696 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/195c489c149f485792a3e9c1223f8bfe as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe 2023-03-24 21:55:41,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe, entries=300, sequenceid=54, filesize=18.5 K 2023-03-24 21:55:41,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/010b16c47fad42b698ae9620478b3750 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750 2023-03-24 21:55:41,705 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750, entries=150, sequenceid=54, filesize=11.6 K 2023-03-24 21:55:41,706 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/34663e532a0048528e6c15f7d43ce27d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d 2023-03-24 21:55:41,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d, entries=150, sequenceid=54, filesize=11.6 K 2023-03-24 21:55:41,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2a79b669e50783c6f283e5def38723af in 138ms, sequenceid=54, compaction requested=true 2023-03-24 21:55:41,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:41,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:41,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:41,712 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:41,712 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:41,712 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:41,713 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:41,713 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:41,713 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:41,713 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:41,714 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,714 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,714 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.8 K 2023-03-24 21:55:41,714 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=41.7 K 2023-03-24 21:55:41,714 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting dfad0585b41b4f318423845439d89de7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679694940407 2023-03-24 21:55:41,714 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 95dea600d70e437dbdcd2f57d7508e51, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679694940407 2023-03-24 21:55:41,715 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 673bd2f68d0944aebdb3a523189728eb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1679694940434 2023-03-24 21:55:41,715 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 37d94f94ed9b47739682e1f7085fe16a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1679694940434 2023-03-24 21:55:41,716 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 195c489c149f485792a3e9c1223f8bfe, keycount=300, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941569 2023-03-24 21:55:41,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:41,716 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:55:41,716 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:41,716 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:41,716 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 010b16c47fad42b698ae9620478b3750, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941572 2023-03-24 21:55:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:41,717 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:41,726 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#387 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:41,727 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#388 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:41,757 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/8469ed95fdf34189a78d383a96c0a225 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8469ed95fdf34189a78d383a96c0a225 2023-03-24 21:55:41,764 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 8469ed95fdf34189a78d383a96c0a225(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:41,764 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:41,764 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694941712; duration=0sec 2023-03-24 21:55:41,764 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:41,764 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:41,765 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:41,765 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:41,766 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,766 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.8 K 2023-03-24 21:55:41,766 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e0fa8904fa4e4c4aa65117d2dad5cbcd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679694940407 2023-03-24 21:55:41,766 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ad68c47ac8d74432b3d65dcbaceea407, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=43, earliestPutTs=1679694940434 2023-03-24 21:55:41,767 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 34663e532a0048528e6c15f7d43ce27d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941572 2023-03-24 21:55:41,771 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/1a70e6f2de7f48b783faa78711dd99e7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1a70e6f2de7f48b783faa78711dd99e7 2023-03-24 21:55:41,776 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#389 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:41,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695001765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695001766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695001767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,778 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 1a70e6f2de7f48b783faa78711dd99e7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:41,778 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:41,778 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694941712; duration=0sec 2023-03-24 21:55:41,778 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:41,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695001771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,785 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695001779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,789 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:41,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:41,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:41,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:41,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:41,793 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:41,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:41,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:41,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,793 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,795 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:41,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:41,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,796 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:41,797 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:41,797 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:41,797 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:41,798 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,798 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:41,798 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:41,798 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:41,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:41,799 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,799 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,800 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,800 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:41,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e7e73f8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:41,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:41,801 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:41,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:41,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:41,803 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:41,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:41,804 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,804 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:41,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:41,806 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:41,806 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:41,806 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:41,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:41,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:41,807 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:41,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,816 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:41,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,816 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,817 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:41,817 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:41,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,880 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695001878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695001881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695001886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,917 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:41,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:41,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:41,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:41,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:41,920 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:41,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:41,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:41,921 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,921 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,923 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:41,923 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:41,923 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:41,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:41,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:41,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:41,924 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,924 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:41,924 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:41,924 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:41,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:41,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,925 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,926 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,926 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:41,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15d955f8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:41,927 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:41,927 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:41,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:41,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:41,930 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:41,931 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:41,931 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,931 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,931 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:41,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:41,933 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:41,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:41,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:41,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:41,934 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:41,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,935 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,935 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:41,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:41,937 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:41,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:41,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:41,939 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:41,939 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:41,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:41,939 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,940 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:41,940 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,940 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:41,940 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:41,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695001979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:41,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:41,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695001979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,040 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,043 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,048 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,048 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,049 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,049 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,049 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b852bd9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,052 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,052 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,053 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,054 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,054 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,054 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,058 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,058 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,058 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,060 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,065 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,065 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,065 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695002082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695002085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695002092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/b91a8fbf9a13434b9711aedab7e64e0a 2023-03-24 21:55:42,159 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/8aa7944435f049978759b18f3a2640cb 2023-03-24 21:55:42,165 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,176 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,177 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,177 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,179 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,187 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,187 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,187 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,191 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,191 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,191 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,191 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,192 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=91 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/2792ec870f89415388e0755df3711533 2023-03-24 21:55:42,193 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,193 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,193 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,194 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1680770b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,197 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/11e7cc040d704711b96ace26e5d98833 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/11e7cc040d704711b96ace26e5d98833 2023-03-24 21:55:42,197 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,198 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,198 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,198 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/b91a8fbf9a13434b9711aedab7e64e0a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a 2023-03-24 21:55:42,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,200 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,200 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,202 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 11e7cc040d704711b96ace26e5d98833(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:42,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,202 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:42,202 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694941712; duration=0sec 2023-03-24 21:55:42,202 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,203 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,204 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,206 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,206 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,206 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,207 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,208 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a, entries=200, sequenceid=91, filesize=13.9 K 2023-03-24 21:55:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/8aa7944435f049978759b18f3a2640cb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb 2023-03-24 21:55:42,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,213 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb, entries=150, sequenceid=91, filesize=11.6 K 2023-03-24 21:55:42,214 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/2792ec870f89415388e0755df3711533 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533 2023-03-24 21:55:42,214 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,214 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,214 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,214 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,215 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:42,215 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,215 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,218 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533, entries=150, sequenceid=91, filesize=11.6 K 2023-03-24 21:55:42,219 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 2a79b669e50783c6f283e5def38723af in 503ms, sequenceid=91, compaction requested=false 2023-03-24 21:55:42,219 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:42,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:42,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:42,287 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:42,315 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,317 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,318 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,321 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,321 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,324 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,324 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,324 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,324 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,324 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,326 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,326 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,327 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,327 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7c2790[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,328 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,329 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,332 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,332 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,332 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,332 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,333 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,333 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,333 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,334 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,336 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,336 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,336 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,338 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,338 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,341 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,341 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,341 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,341 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,342 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:42,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,342 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,342 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694942310, max timestamp=1679694942345Num uniques 859; 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694942310, max timestamp=1679694942345Num uniques 859; 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694942310, max timestamp=1679694942345Num uniques 859; 2023-03-24 21:55:42,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:42,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695002391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695002393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695002394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695002397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695002399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,443 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,446 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,449 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,451 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,452 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,453 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54d5c592[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,459 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,459 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,460 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,462 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,462 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,464 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,464 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,464 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,465 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,467 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,467 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,468 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,468 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,468 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:42,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695002500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695002506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695002513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,568 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,570 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,571 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,576 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,576 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,576 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,576 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65ce732[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,581 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,581 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,586 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,586 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,586 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,588 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,589 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:42,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,599 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,599 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,599 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,693 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/87e53ed674dc4fddbf58daf19ae053ab 2023-03-24 21:55:42,699 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,702 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,703 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/c823122e496a427da7ded49ec2b1b2e5 2023-03-24 21:55:42,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695002704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,707 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,707 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,708 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,708 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,709 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,709 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,709 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,709 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,711 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,713 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1371e0dd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,718 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,718 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,719 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695002715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,720 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,720 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,720 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695002723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,729 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,729 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,729 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,731 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,732 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,744 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,744 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,744 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,744 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,745 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,745 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,745 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,845 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,848 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,849 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,851 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,853 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,853 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,853 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,853 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:42,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79db1584[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:42,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:42,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:42,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:42,859 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:42,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:42,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:42,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,860 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:42,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:42,862 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:42,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:42,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:42,862 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:42,862 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:42,864 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,864 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,864 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,864 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,867 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:42,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,871 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:42,871 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:42,871 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695002900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:42,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695002908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,971 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:42,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:42,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:42,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:42,979 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:42,979 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,979 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:42,979 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:42,981 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,981 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:42,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:42,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:42,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:42,984 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:42,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:42,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:42,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:42,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:42,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:42,995 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:42,995 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:42,996 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:42,996 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,996 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:42,996 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:42,997 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,997 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:42,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:42,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:42,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:42,998 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:42,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:43,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@358b678[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:43,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:43,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:43,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:43,001 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:43,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:43,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:43,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:43,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:43,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:43,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:43,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:43,007 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:43,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:43,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:43,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:43,007 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:43,008 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:43,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:43,008 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,008 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:43,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:43,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:43,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695003010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,014 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:43,014 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:43,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:43,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:43,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:43,015 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:43,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:43,015 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:43,015 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:43,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:43,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:43,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695003021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695003028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,115 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:43,116 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/dde3df054da74eb9984a982d7e2ca640 2023-03-24 21:55:43,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:43,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:43,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:43,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:43,118 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:43,118 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:43,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:43,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:43,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,120 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/87e53ed674dc4fddbf58daf19ae053ab as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab 2023-03-24 21:55:43,120 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:43,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:43,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:43,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:43,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:43,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:43,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:43,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:43,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:43,122 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:43,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:43,123 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab, entries=200, sequenceid=105, filesize=13.9 K 2023-03-24 21:55:43,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/c823122e496a427da7ded49ec2b1b2e5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5 2023-03-24 21:55:43,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5, entries=150, sequenceid=105, filesize=11.6 K 2023-03-24 21:55:43,127 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:43,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:43,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,128 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:43,128 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/dde3df054da74eb9984a982d7e2ca640 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640 2023-03-24 21:55:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:43,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:43,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625ca456[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:43,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:43,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:43,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:43,130 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:43,131 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:43,131 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640, entries=150, sequenceid=105, filesize=11.6 K 2023-03-24 21:55:43,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:43,132 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:43,132 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2a79b669e50783c6f283e5def38723af in 846ms, sequenceid=105, compaction requested=true 2023-03-24 21:55:43,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:43,132 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:43,132 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:43,132 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:43,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,133 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:43,133 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:43,133 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:43,133 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,133 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:43,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,133 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,133 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1a70e6f2de7f48b783faa78711dd99e7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=39.5 K 2023-03-24 21:55:43,133 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8469ed95fdf34189a78d383a96c0a225, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.9 K 2023-03-24 21:55:43,134 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1a70e6f2de7f48b783faa78711dd99e7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941572 2023-03-24 21:55:43,134 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8469ed95fdf34189a78d383a96c0a225, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941572 2023-03-24 21:55:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,134 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b91a8fbf9a13434b9711aedab7e64e0a, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679694941638 2023-03-24 21:55:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,134 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8aa7944435f049978759b18f3a2640cb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679694941649 2023-03-24 21:55:43,134 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 87e53ed674dc4fddbf58daf19ae053ab, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941737 2023-03-24 21:55:43,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,135 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c823122e496a427da7ded49ec2b1b2e5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941769 2023-03-24 21:55:43,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:43,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:43,135 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:43,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:43,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:43,135 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:43,135 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:43,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:43,138 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:43,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:43,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:43,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:43,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:43,142 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#395 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:43,142 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#396 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:43,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:43,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:43,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:43,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:43,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,145 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:43,145 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:43,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:43,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:43,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:43,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:43,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,161 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/1d1bd750d501493a8d32da93a3a22058 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/1d1bd750d501493a8d32da93a3a22058 2023-03-24 21:55:43,164 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 1d1bd750d501493a8d32da93a3a22058(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:43,164 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:43,164 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694943132; duration=0sec 2023-03-24 21:55:43,164 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:43,164 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:43,165 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:43,165 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:43,165 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,165 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/11e7cc040d704711b96ace26e5d98833, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.9 K 2023-03-24 21:55:43,167 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 11e7cc040d704711b96ace26e5d98833, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=54, earliestPutTs=1679694941572 2023-03-24 21:55:43,167 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2792ec870f89415388e0755df3711533, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=91, earliestPutTs=1679694941649 2023-03-24 21:55:43,168 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting dde3df054da74eb9984a982d7e2ca640, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941769 2023-03-24 21:55:43,173 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#397 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:43,245 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:43,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:43,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:43,249 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:55:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:43,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:43,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:43,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:43,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:43,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:43,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:43,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:43,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:43,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:43,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:43,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:43,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:43,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:43,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:43,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:43,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:43,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:43,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:43,254 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:43,254 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:43,255 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:43,255 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:43,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:43,517 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:43,542 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695003541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,547 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695003542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,548 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695003543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,563 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/6b0e7e4c166b44f2967aa1729ac96293 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6b0e7e4c166b44f2967aa1729ac96293 2023-03-24 21:55:43,566 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 6b0e7e4c166b44f2967aa1729ac96293(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:43,567 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:43,567 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694943132; duration=0sec 2023-03-24 21:55:43,567 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:43,581 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/0843f8f81415411faf73ae7076df73ec as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/0843f8f81415411faf73ae7076df73ec 2023-03-24 21:55:43,585 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 0843f8f81415411faf73ae7076df73ec(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:43,585 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:43,585 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694943132; duration=0sec 2023-03-24 21:55:43,585 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:43,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695003644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695003648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695003649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,669 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/abf86d959d024ee28e05cffd8ee49e2d 2023-03-24 21:55:43,855 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695003850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695003852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695003853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695003909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:43,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:43,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695003921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,080 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/9e5c44bec3b7411cb6fda2a183e296c3 2023-03-24 21:55:44,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695004157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695004159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695004160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,491 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=141 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/121e08498fba4958bf064a56c8f20093 2023-03-24 21:55:44,494 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/abf86d959d024ee28e05cffd8ee49e2d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d 2023-03-24 21:55:44,497 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d, entries=150, sequenceid=141, filesize=11.8 K 2023-03-24 21:55:44,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/9e5c44bec3b7411cb6fda2a183e296c3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3 2023-03-24 21:55:44,501 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3, entries=150, sequenceid=141, filesize=11.8 K 2023-03-24 21:55:44,502 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/121e08498fba4958bf064a56c8f20093 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093 2023-03-24 21:55:44,505 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093, entries=150, sequenceid=141, filesize=11.8 K 2023-03-24 21:55:44,506 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2a79b669e50783c6f283e5def38723af in 1252ms, sequenceid=141, compaction requested=false 2023-03-24 21:55:44,506 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:44,506 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,506 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:44,506 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:44,506 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,506 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:44,506 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:44,507 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:44,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e37d7c8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:44,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:44,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,511 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:44,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:44,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:44,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:44,512 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:44,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:44,515 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:44,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:44,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:44,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:44,515 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:44,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:44,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,521 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,521 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:44,521 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:44,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,621 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:44,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:44,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:44,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:44,624 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:44,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:44,625 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:44,625 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,625 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,627 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,627 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:44,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:44,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,628 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:44,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:44,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:44,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:44,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:44,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:44,629 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:44,629 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,629 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:44,629 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:44,629 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:44,630 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,630 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:44,630 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,630 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:44,630 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,637 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/cdd087ef5dda4c85bbcda9c6979f62d1 2023-03-24 21:55:44,648 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/c97b3d197b944b21bd45a137be20c987 2023-03-24 21:55:44,658 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=154 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/a3d0c04e0ae84e8080f952d64eab301c 2023-03-24 21:55:44,661 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/cdd087ef5dda4c85bbcda9c6979f62d1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1 2023-03-24 21:55:44,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:44,662 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:44,664 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1, entries=150, sequenceid=154, filesize=11.8 K 2023-03-24 21:55:44,665 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/c97b3d197b944b21bd45a137be20c987 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987 2023-03-24 21:55:44,668 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987, entries=150, sequenceid=154, filesize=11.8 K 2023-03-24 21:55:44,669 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/a3d0c04e0ae84e8080f952d64eab301c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c 2023-03-24 21:55:44,672 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c, entries=150, sequenceid=154, filesize=11.8 K 2023-03-24 21:55:44,672 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=40.25 KB/41220 for 2a79b669e50783c6f283e5def38723af in 43ms, sequenceid=154, compaction requested=true 2023-03-24 21:55:44,672 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:44,672 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,672 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:44,673 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:44,673 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,673 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:44,673 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:44,674 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:44,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:44,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:44,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,675 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:44,675 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:44,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:44,676 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe19abf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:44,676 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:44,676 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,678 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,678 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:44,678 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:44,678 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:44,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,679 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:44,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:44,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,680 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:44,680 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:44,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:44,682 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:44,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:44,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:44,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:44,683 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:44,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:44,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:44,690 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,691 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:44,691 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:44,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:44,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694944704, max timestamp=1679694944714Num uniques 859; 2023-03-24 21:55:44,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:44,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694944704, max timestamp=1679694944714Num uniques 859; 2023-03-24 21:55:44,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694944704, max timestamp=1679694944714Num uniques 859; 2023-03-24 21:55:44,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:44,717 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:44,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695004759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695004760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695004760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,791 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:44,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:44,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:44,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:44,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:44,807 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:44,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:44,807 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:44,808 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,808 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,809 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,809 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:44,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:44,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:44,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:44,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:44,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:44,811 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:44,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:44,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:44,812 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:44,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:44,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:44,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:44,813 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:44,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19cc1c4d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:44,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:44,819 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:44,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:44,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:44,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:44,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,826 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:44,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:44,829 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:44,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:44,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:44,829 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:44,829 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:44,831 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,831 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,831 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,838 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:44,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,838 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:44,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,839 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:44,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:44,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695004869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695004870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:44,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695004872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,939 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:44,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:44,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:44,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:44,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:44,942 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:44,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:44,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:44,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,943 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,944 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,944 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:44,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:44,946 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:44,946 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:44,946 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,946 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:44,946 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:44,946 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:44,946 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:44,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:44,947 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:44,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,949 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:44,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e357536[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:44,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:44,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,951 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:44,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:44,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:44,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,952 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:44,954 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,954 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:44,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,955 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:44,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:44,957 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:44,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:44,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:44,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:44,957 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:44,958 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,958 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,958 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:44,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:44,959 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,960 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:44,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,962 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,963 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:44,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:44,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:44,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:44,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:44,963 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:44,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:44,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:44,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:44,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:44,974 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:44,974 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:44,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:44,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,074 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:45,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:45,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695005072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,076 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695005073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:45,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:45,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:45,077 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:45,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:45,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:45,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/629c76a26a6d40a298cbb848e59c3e1f 2023-03-24 21:55:45,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,081 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,082 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:45,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:45,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:45,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:45,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:45,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:45,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:45,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:45,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:45,083 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:45,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:45,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:45,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:45,089 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,089 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,090 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,090 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695005085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,091 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/6ac0efc6da5d4568b5c83e661cf929cc 2023-03-24 21:55:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:45,092 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7c78af[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:45,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:45,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,093 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:45,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:45,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:45,094 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,094 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:45,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:45,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:45,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:45,099 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:45,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:45,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:45,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:45,099 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:45,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,102 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,103 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:45,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,108 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:45,108 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,208 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:45,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:45,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:45,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:45,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:45,212 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:45,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:45,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:45,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,212 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,213 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,213 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,214 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:45,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:45,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:45,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:45,215 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,215 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:45,215 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:45,215 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:45,215 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:45,216 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,216 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:45,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9b25bd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:45,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:45,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,218 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:45,219 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,219 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:45,219 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:45,219 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:45,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:45,224 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,224 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:45,225 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,225 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:45,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:45,227 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:45,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:45,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:45,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:45,227 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:45,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,228 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,229 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:45,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:45,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,233 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,233 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:45,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,234 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,234 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,234 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,333 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:45,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:45,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:45,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:45,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:45,337 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:45,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:45,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:45,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,337 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,339 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,339 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,340 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,340 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:45,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:45,341 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:45,341 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:45,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:45,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:45,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:45,341 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:45,341 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,341 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:45,342 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:45,342 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:45,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:45,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:45,342 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:45,343 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,343 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,344 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,344 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:45,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74cc6e37[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:45,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:45,345 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,346 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:45,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:45,347 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:45,347 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,347 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:45,348 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:45,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,348 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:45,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:45,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:45,352 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:45,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:45,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:45,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:45,352 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:45,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,358 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:45,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,362 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:45,362 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:45,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:45,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695005378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695005379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695005393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,463 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:45,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:45,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:45,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:45,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:45,466 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:45,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:45,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:45,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,467 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:45,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:45,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:45,468 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:45,469 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,469 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:45,469 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:45,469 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:45,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:45,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,470 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,471 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,471 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:45,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4165295d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:45,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:45,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,475 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,475 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:45,475 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:45,475 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:45,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:45,477 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:45,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,477 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:45,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:45,479 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:45,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:45,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:45,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:45,479 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:45,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:45,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:45,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:45,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:45,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:45,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:45,485 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:45,485 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:45,484 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,485 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:45,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:45,485 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,485 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:45,485 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,501 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=164 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/973245f177ff4941818230109b3addc4 2023-03-24 21:55:45,504 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/629c76a26a6d40a298cbb848e59c3e1f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f 2023-03-24 21:55:45,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f, entries=200, sequenceid=164, filesize=14.1 K 2023-03-24 21:55:45,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/6ac0efc6da5d4568b5c83e661cf929cc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc 2023-03-24 21:55:45,510 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc, entries=150, sequenceid=164, filesize=11.8 K 2023-03-24 21:55:45,511 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/973245f177ff4941818230109b3addc4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4 2023-03-24 21:55:45,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4, entries=150, sequenceid=164, filesize=11.8 K 2023-03-24 21:55:45,515 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2a79b669e50783c6f283e5def38723af in 840ms, sequenceid=164, compaction requested=true 2023-03-24 21:55:45,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:45,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:45,515 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:45,515 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:45,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:45,515 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:45,516 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50590 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:45,516 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:45,516 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:45,516 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:45,516 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,516 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,516 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6b0e7e4c166b44f2967aa1729ac96293, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=49.4 K 2023-03-24 21:55:45,516 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/1d1bd750d501493a8d32da93a3a22058, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=47.1 K 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6b0e7e4c166b44f2967aa1729ac96293, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941769 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1d1bd750d501493a8d32da93a3a22058, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941769 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting abf86d959d024ee28e05cffd8ee49e2d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679694942384 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9e5c44bec3b7411cb6fda2a183e296c3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679694942384 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cdd087ef5dda4c85bbcda9c6979f62d1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679694943528 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c97b3d197b944b21bd45a137be20c987, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679694943528 2023-03-24 21:55:45,517 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 629c76a26a6d40a298cbb848e59c3e1f, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:45,518 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6ac0efc6da5d4568b5c83e661cf929cc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:45,523 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#407 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:45,523 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#408 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:45,531 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5acff26721e94e4d8599cb596188bdbc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5acff26721e94e4d8599cb596188bdbc 2023-03-24 21:55:45,534 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 5acff26721e94e4d8599cb596188bdbc(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:45,535 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:45,535 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=12, startTime=1679694945515; duration=0sec 2023-03-24 21:55:45,535 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:45,535 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:45,536 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:45,537 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:45,537 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,537 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/0843f8f81415411faf73ae7076df73ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=47.1 K 2023-03-24 21:55:45,537 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0843f8f81415411faf73ae7076df73ec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679694941769 2023-03-24 21:55:45,537 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 121e08498fba4958bf064a56c8f20093, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=141, earliestPutTs=1679694942384 2023-03-24 21:55:45,537 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a3d0c04e0ae84e8080f952d64eab301c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=154, earliestPutTs=1679694943528 2023-03-24 21:55:45,538 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 973245f177ff4941818230109b3addc4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:45,542 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#409 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:45,585 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:45,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:45,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:45,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:45,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:45,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:45,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:45,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:45,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,590 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:45,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:45,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:45,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:45,592 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:45,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:45,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:45,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:45,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:45,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:45,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:45,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:45,592 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:45,592 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:45,592 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 21:55:45,592 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:45,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:45,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:45,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:45,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:45,593 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:45,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:45,891 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:45,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695005906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695005907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695005912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,932 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695005922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:45,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695005925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:45,934 DEBUG [Thread-3038] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4162 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:45,939 DEBUG [Thread-3040] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4156 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:45,939 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/b0c5fd81584b425793ccad087da24f96 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b0c5fd81584b425793ccad087da24f96 2023-03-24 21:55:45,948 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into b0c5fd81584b425793ccad087da24f96(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:45,948 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:45,948 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=12, startTime=1679694945515; duration=0sec 2023-03-24 21:55:45,948 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:45,952 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/95cf72811c3742578db21c2270738832 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/95cf72811c3742578db21c2270738832 2023-03-24 21:55:45,959 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 95cf72811c3742578db21c2270738832(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:45,959 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:45,959 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=12, startTime=1679694945515; duration=0sec 2023-03-24 21:55:45,959 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:46,000 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/cba8f88684d141248cc68e2447c34523 2023-03-24 21:55:46,023 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/99a9eab9eb2a4de29347f0fbec5903aa 2023-03-24 21:55:46,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695006020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695006023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,029 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695006024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,037 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/d7aaf9140a924f50ab2fb22a39dedf58 2023-03-24 21:55:46,041 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/cba8f88684d141248cc68e2447c34523 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523 2023-03-24 21:55:46,046 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523, entries=150, sequenceid=201, filesize=11.8 K 2023-03-24 21:55:46,047 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/99a9eab9eb2a4de29347f0fbec5903aa as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa 2023-03-24 21:55:46,051 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa, entries=150, sequenceid=201, filesize=11.8 K 2023-03-24 21:55:46,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/d7aaf9140a924f50ab2fb22a39dedf58 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58 2023-03-24 21:55:46,055 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58, entries=150, sequenceid=201, filesize=11.8 K 2023-03-24 21:55:46,056 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2a79b669e50783c6f283e5def38723af in 464ms, sequenceid=201, compaction requested=false 2023-03-24 21:55:46,056 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:46,056 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:46,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:46,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:46,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:46,056 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:46,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,059 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,059 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:46,062 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:46,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:46,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:46,064 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c6188de[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:46,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:46,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,065 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:46,065 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:46,066 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:46,066 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,066 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:46,067 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:46,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:46,068 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:46,068 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:46,068 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:46,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:46,071 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:46,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:46,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:46,071 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:46,071 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:46,074 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,074 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,074 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:46,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:46,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:46,075 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:46,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:46,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:46,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:46,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:46,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:46,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:46,086 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:46,086 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,087 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:46,087 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:46,187 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:46,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:46,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:46,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:46,193 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:46,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:46,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:46,193 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:46,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:46,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,195 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:46,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:46,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:46,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:46,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:46,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:46,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:46,199 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:46,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:46,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:46,199 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:46,206 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:46,206 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:46,206 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:46,206 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:46,206 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:46,206 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:46,206 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:46,207 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:46,207 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:46,207 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:46,207 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:46,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:46,229 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:46,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694946255, max timestamp=1679694946279Num uniques 859; 2023-03-24 21:55:46,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:46,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694946255, max timestamp=1679694946279Num uniques 859; 2023-03-24 21:55:46,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:46,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694946255, max timestamp=1679694946279Num uniques 859; 2023-03-24 21:55:46,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:46,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695006321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695006322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695006325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695006431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695006431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695006431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,633 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/77652401d79e46628d3f1e20e7a57e0e 2023-03-24 21:55:46,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695006636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695006636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,642 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695006636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,656 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/fe749462862b4361bb3ba2425f8ee546 2023-03-24 21:55:46,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695006945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695006945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:46,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:46,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695006946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,069 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/7ed0ebd39ef54692a81a16f10a89ae02 2023-03-24 21:55:47,073 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/77652401d79e46628d3f1e20e7a57e0e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e 2023-03-24 21:55:47,081 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e, entries=100, sequenceid=213, filesize=9.4 K 2023-03-24 21:55:47,082 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/fe749462862b4361bb3ba2425f8ee546 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546 2023-03-24 21:55:47,089 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546, entries=100, sequenceid=213, filesize=9.4 K 2023-03-24 21:55:47,090 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/7ed0ebd39ef54692a81a16f10a89ae02 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02 2023-03-24 21:55:47,093 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02, entries=100, sequenceid=213, filesize=9.4 K 2023-03-24 21:55:47,094 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2a79b669e50783c6f283e5def38723af in 888ms, sequenceid=213, compaction requested=true 2023-03-24 21:55:47,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:47,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:47,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:47,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:47,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:47,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:47,097 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,097 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:47,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:47,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2883fc22[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:47,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:47,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:47,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:47,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:47,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:47,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:47,102 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:47,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:47,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:47,103 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:47,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:47,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:47,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:47,103 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:47,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:47,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:47,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,114 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:47,114 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:47,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:47,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:47,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:47,115 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:47,115 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:47,215 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:47,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:47,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:47,218 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:47,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:47,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:47,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:47,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:47,220 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:47,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:47,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:47,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:47,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:47,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:47,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:47,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:47,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:47,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:47,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:47,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:47,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:47,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:47,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:47,222 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 21:55:47,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:47,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:47,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:47,223 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:47,223 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:47,223 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:47,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:47,456 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:47,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695007468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695007469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,477 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695007469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695007572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695007577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695007578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,635 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/1b1407e08a6249a2896182f101966e5e 2023-03-24 21:55:47,644 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/45f2c28eecae4791959cacf059c2c898 2023-03-24 21:55:47,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48194 deadline: 1679695007779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48210 deadline: 1679695007785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:47,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:47,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695007785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,053 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/b94bde939e6c4e3d8b13b441c6ca4d2a 2023-03-24 21:55:48,058 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/1b1407e08a6249a2896182f101966e5e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e 2023-03-24 21:55:48,061 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e, entries=150, sequenceid=249, filesize=11.8 K 2023-03-24 21:55:48,062 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/45f2c28eecae4791959cacf059c2c898 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898 2023-03-24 21:55:48,066 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898, entries=150, sequenceid=249, filesize=11.8 K 2023-03-24 21:55:48,067 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/b94bde939e6c4e3d8b13b441c6ca4d2a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a 2023-03-24 21:55:48,071 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a, entries=150, sequenceid=249, filesize=11.8 K 2023-03-24 21:55:48,072 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2a79b669e50783c6f283e5def38723af in 850ms, sequenceid=249, compaction requested=true 2023-03-24 21:55:48,072 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,072 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,072 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,074 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,074 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,075 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa992d3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,078 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,080 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,080 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,083 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,083 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,083 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,084 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:48,085 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:48,085 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:48,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,088 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/11972ed817e04606a68d7e90cf6c108c 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,104 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,104 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,113 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/985c44d9910046dab748bab4fb2e2345 2023-03-24 21:55:48,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ee7395e4a2de4dc58a6f861a466b941f 2023-03-24 21:55:48,135 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/11972ed817e04606a68d7e90cf6c108c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c 2023-03-24 21:55:48,138 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694948123, max timestamp=1679694948137Num uniques 859; 2023-03-24 21:55:48,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694948123, max timestamp=1679694948137Num uniques 859; 2023-03-24 21:55:48,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694948123, max timestamp=1679694948137Num uniques 859; 2023-03-24 21:55:48,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,139 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c, entries=150, sequenceid=260, filesize=11.8 K 2023-03-24 21:55:48,141 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/985c44d9910046dab748bab4fb2e2345 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345 2023-03-24 21:55:48,145 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345, entries=100, sequenceid=260, filesize=9.5 K 2023-03-24 21:55:48,145 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ee7395e4a2de4dc58a6f861a466b941f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f 2023-03-24 21:55:48,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f, entries=100, sequenceid=260, filesize=9.5 K 2023-03-24 21:55:48,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=53.67 KB/54960 for 2a79b669e50783c6f283e5def38723af in 64ms, sequenceid=260, compaction requested=true 2023-03-24 21:55:48,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,149 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:48,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:48,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:48,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:48,150 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:48,152 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58224 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:48,152 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:48,152 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,152 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b0c5fd81584b425793ccad087da24f96, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=56.9 K 2023-03-24 21:55:48,152 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b0c5fd81584b425793ccad087da24f96, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:48,152 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=67.09 KB heapSize=181.08 KB 2023-03-24 21:55:48,152 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55784 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:48,153 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cba8f88684d141248cc68e2447c34523, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679694944743 2023-03-24 21:55:48,153 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:48,153 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,153 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5acff26721e94e4d8599cb596188bdbc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=54.5 K 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:48,153 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 77652401d79e46628d3f1e20e7a57e0e, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679694945903 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:48,153 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,153 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5acff26721e94e4d8599cb596188bdbc, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:48,153 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1b1407e08a6249a2896182f101966e5e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1679694946319 2023-03-24 21:55:48,154 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 99a9eab9eb2a4de29347f0fbec5903aa, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679694944743 2023-03-24 21:55:48,154 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 11972ed817e04606a68d7e90cf6c108c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694947467 2023-03-24 21:55:48,154 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting fe749462862b4361bb3ba2425f8ee546, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679694945903 2023-03-24 21:55:48,156 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 45f2c28eecae4791959cacf059c2c898, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1679694946319 2023-03-24 21:55:48,157 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 985c44d9910046dab748bab4fb2e2345, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694947467 2023-03-24 21:55:48,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d92ff826161e49a996f94ab28f56536a 2023-03-24 21:55:48,166 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#423 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:48,169 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#424 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:48,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/4f0141922ad24d9aad6c3c6f8c3841a2 2023-03-24 21:55:48,181 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/f03d072722da475f85d3cab404a44dc1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/f03d072722da475f85d3cab404a44dc1 2023-03-24 21:55:48,185 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into f03d072722da475f85d3cab404a44dc1(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:48,185 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,185 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=11, startTime=1679694948150; duration=0sec 2023-03-24 21:55:48,185 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:48,186 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:48,197 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 55784 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:48,197 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:48,198 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,198 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/95cf72811c3742578db21c2270738832, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=54.5 K 2023-03-24 21:55:48,198 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 95cf72811c3742578db21c2270738832, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=164, earliestPutTs=1679694944662 2023-03-24 21:55:48,198 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d7aaf9140a924f50ab2fb22a39dedf58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1679694944743 2023-03-24 21:55:48,199 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7ed0ebd39ef54692a81a16f10a89ae02, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1679694945903 2023-03-24 21:55:48,199 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b94bde939e6c4e3d8b13b441c6ca4d2a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1679694946319 2023-03-24 21:55:48,199 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ee7395e4a2de4dc58a6f861a466b941f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694947467 2023-03-24 21:55:48,205 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,215 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,215 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,217 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,219 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679694948179, max timestamp=1679694948219Num uniques 859; 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679694948179, max timestamp=1679694948219Num uniques 859; 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679694948179, max timestamp=1679694948219Num uniques 859; 2023-03-24 21:55:48,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:48,229 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#427 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:48,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:48,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,243 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,246 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576e6bba[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,254 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,254 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,267 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,267 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,294 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695008284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695008285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695008285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,367 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,371 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,372 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,373 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:48,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,375 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,382 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,383 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,384 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27393ef1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,389 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,389 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,389 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,396 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,396 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,399 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,401 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,403 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:48,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,404 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,404 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,404 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48142 deadline: 1679695008395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695008403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695008403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,504 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,507 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:48,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,512 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,512 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,512 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,512 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,512 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,514 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,514 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,514 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,514 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106cb4d8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,516 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,520 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,520 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,525 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,526 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,527 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,527 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,582 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/1fd29c8cd97d4679904309abca4d6398 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1fd29c8cd97d4679904309abca4d6398 2023-03-24 21:55:48,585 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 1fd29c8cd97d4679904309abca4d6398(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:48,586 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,586 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=11, startTime=1679694948149; duration=0sec 2023-03-24 21:55:48,586 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:48,598 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=286 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/33694e288c5544108ab4e09e8afed037 2023-03-24 21:55:48,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d92ff826161e49a996f94ab28f56536a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a 2023-03-24 21:55:48,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a, entries=300, sequenceid=286, filesize=19.0 K 2023-03-24 21:55:48,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/4f0141922ad24d9aad6c3c6f8c3841a2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2 2023-03-24 21:55:48,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2, entries=150, sequenceid=286, filesize=11.9 K 2023-03-24 21:55:48,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/33694e288c5544108ab4e09e8afed037 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037 2023-03-24 21:55:48,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037, entries=150, sequenceid=286, filesize=11.9 K 2023-03-24 21:55:48,612 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~181.03 KB/185376, currentSize=120.76 KB/123660 for 2a79b669e50783c6f283e5def38723af in 460ms, sequenceid=286, compaction requested=false 2023-03-24 21:55:48,612 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:48,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:48,614 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:48,627 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,630 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,632 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:48,634 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,634 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,634 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,636 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a0830c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695008632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695008637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,641 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,644 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,644 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695008641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:48,652 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,652 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,652 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,670 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/6c495d15fc47446fb1ab2207678266c9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6c495d15fc47446fb1ab2207678266c9 2023-03-24 21:55:48,673 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 6c495d15fc47446fb1ab2207678266c9(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:48,674 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:48,674 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=11, startTime=1679694948150; duration=0sec 2023-03-24 21:55:48,674 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:48,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695008742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,747 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695008742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,753 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695008751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,756 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,756 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:48,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,764 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,764 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,764 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,764 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,766 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47b71779[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,768 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,769 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,770 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,770 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,770 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,770 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,772 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,772 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,775 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,775 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,775 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,777 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,780 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,780 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,780 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,780 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,880 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:48,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:48,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:48,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:48,884 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:48,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:48,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:48,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,886 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,887 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,887 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:48,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:48,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:48,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:48,890 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,890 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:48,890 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:48,890 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,890 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:48,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:48,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,897 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:48,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@299777c8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:48,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:48,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:48,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:48,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,901 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:48,902 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:48,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:48,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:48,905 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:48,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:48,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:48,905 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:48,905 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:48,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:48,907 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:48,908 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:48,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:48,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:48,915 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:48,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:48,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:48,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:48,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:48,915 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:48,915 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:48,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695008948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695008949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:48,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:48,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695008954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,016 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,019 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,020 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,024 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,025 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c28cc56[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,028 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,029 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d46291d816e6413f822890de8dc3e82d 2023-03-24 21:55:49,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,034 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,041 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,041 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:49,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,046 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,047 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,047 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,047 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,147 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,150 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,150 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,152 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,154 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,154 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,154 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,154 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,156 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b779c3c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,164 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,164 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,167 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,167 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,167 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,172 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,173 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,173 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,173 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,258 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695009254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695009254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695009262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,273 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,276 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,276 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,277 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,278 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,278 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,279 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,280 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,280 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,280 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,280 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,282 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@440458f4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,284 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,285 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,287 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,288 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,296 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,296 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,300 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,302 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:49,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,303 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,303 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,403 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,406 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,406 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,407 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,409 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,411 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,412 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,413 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,413 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56bf5e8b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,417 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,425 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,428 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,428 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,428 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,431 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,438 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,438 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:49,438 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,439 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/938dfaa82f83495b90c01d2851eaa952 2023-03-24 21:55:49,539 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,542 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,543 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,544 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,545 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,548 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,548 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,548 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,548 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cef6276[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,556 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,558 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,558 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,560 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:49,564 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,564 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,564 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,664 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,668 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,670 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,673 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,673 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,675 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,679 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,682 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,682 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b7e81ab[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,686 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,686 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,687 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,691 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,691 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,692 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,692 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,699 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,699 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695009762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695009763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:49,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695009773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,799 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,803 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,805 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,805 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,806 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,806 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,806 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,806 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:49,807 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:49,807 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:49,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,809 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,809 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:49,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d5bb3c8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:49,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:49,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:49,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:49,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:49,812 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:49,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:49,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,814 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:49,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:49,818 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:49,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:49,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:49,818 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:49,818 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:49,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,820 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:49,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,822 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:49,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:49,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:49,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:49,825 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:49,825 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:49,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:49,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:49,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:49,826 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,858 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/91451afa96fb4933b6faf3ccb1a1de50 2023-03-24 21:55:49,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d46291d816e6413f822890de8dc3e82d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d 2023-03-24 21:55:49,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d, entries=200, sequenceid=323, filesize=14.3 K 2023-03-24 21:55:49,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/938dfaa82f83495b90c01d2851eaa952 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952 2023-03-24 21:55:49,869 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952, entries=150, sequenceid=323, filesize=11.9 K 2023-03-24 21:55:49,870 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/91451afa96fb4933b6faf3ccb1a1de50 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50 2023-03-24 21:55:49,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50, entries=150, sequenceid=323, filesize=11.9 K 2023-03-24 21:55:49,874 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2a79b669e50783c6f283e5def38723af in 1260ms, sequenceid=323, compaction requested=true 2023-03-24 21:55:49,874 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:49,874 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:49,875 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:49,875 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:49,875 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:49,875 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 46730 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:49,876 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:49,876 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1fd29c8cd97d4679904309abca4d6398, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=45.6 K 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:49,876 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,876 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/f03d072722da475f85d3cab404a44dc1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.1 K 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1fd29c8cd97d4679904309abca4d6398, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694946324 2023-03-24 21:55:49,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f03d072722da475f85d3cab404a44dc1, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694946324 2023-03-24 21:55:49,877 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d92ff826161e49a996f94ab28f56536a, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679694948126 2023-03-24 21:55:49,877 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4f0141922ad24d9aad6c3c6f8c3841a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679694948130 2023-03-24 21:55:49,877 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d46291d816e6413f822890de8dc3e82d, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:49,877 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 938dfaa82f83495b90c01d2851eaa952, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:49,884 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#431 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:49,888 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#432 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:49,899 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2974f7b2babe47069127f5e7c9985a9b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2974f7b2babe47069127f5e7c9985a9b 2023-03-24 21:55:49,903 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 2974f7b2babe47069127f5e7c9985a9b(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:49,903 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:49,903 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694949875; duration=0sec 2023-03-24 21:55:49,903 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:49,903 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:49,905 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36970 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:49,905 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:49,905 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,905 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6c495d15fc47446fb1ab2207678266c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.1 K 2023-03-24 21:55:49,905 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6c495d15fc47446fb1ab2207678266c9, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694946324 2023-03-24 21:55:49,907 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 33694e288c5544108ab4e09e8afed037, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=286, earliestPutTs=1679694948130 2023-03-24 21:55:49,907 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 91451afa96fb4933b6faf3ccb1a1de50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:49,914 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#433 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:49,924 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/b37851735c67425f9f66460c3f269a24 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b37851735c67425f9f66460c3f269a24 2023-03-24 21:55:49,925 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:49,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:49,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:49,930 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:49,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into b37851735c67425f9f66460c3f269a24(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:49,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:49,930 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:49,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:49,930 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694949875; duration=0sec 2023-03-24 21:55:49,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:49,931 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,931 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:49,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:49,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:49,932 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:49,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:49,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:49,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:49,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:49,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:49,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:49,934 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:49,934 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:49,934 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:49,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:49,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:50,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694949999, max timestamp=1679694950014Num uniques 859; 2023-03-24 21:55:50,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:50,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694949999, max timestamp=1679694950014Num uniques 859; 2023-03-24 21:55:50,015 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:50,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694949999, max timestamp=1679694950014Num uniques 859; 2023-03-24 21:55:50,016 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695010059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695010059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695010166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695010166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,293 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/01c2e2a98e314abcb9904f602c9131a2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/01c2e2a98e314abcb9904f602c9131a2 2023-03-24 21:55:50,304 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 01c2e2a98e314abcb9904f602c9131a2(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:50,304 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:50,304 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694949874; duration=0sec 2023-03-24 21:55:50,305 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:50,341 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/8f604e62efc842ada15f0ebc4fde70fd 2023-03-24 21:55:50,356 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/e19d96fc381c4229837d6631b64cd486 2023-03-24 21:55:50,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695010371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695010372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,682 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695010677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695010678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,769 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=337 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/260c8de726d0424f9d185d815932164d 2023-03-24 21:55:50,773 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/8f604e62efc842ada15f0ebc4fde70fd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd 2023-03-24 21:55:50,778 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd, entries=150, sequenceid=337, filesize=11.9 K 2023-03-24 21:55:50,779 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/e19d96fc381c4229837d6631b64cd486 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486 2023-03-24 21:55:50,783 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486, entries=150, sequenceid=337, filesize=11.9 K 2023-03-24 21:55:50,784 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/260c8de726d0424f9d185d815932164d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d 2023-03-24 21:55:50,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695010778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,787 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d, entries=150, sequenceid=337, filesize=11.9 K 2023-03-24 21:55:50,788 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2a79b669e50783c6f283e5def38723af in 854ms, sequenceid=337, compaction requested=false 2023-03-24 21:55:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:50,788 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:50,788 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:50,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:50,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-24 21:55:50,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:50,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:50,790 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:50,790 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:50,791 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:50,791 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:50,791 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,791 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,792 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,792 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:50,794 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bcebee[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:50,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:50,794 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,795 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,796 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:50,796 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:50,796 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:50,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,797 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:50,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:50,798 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:50,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:50,800 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:50,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:50,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:50,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:50,800 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:50,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:50,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,803 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,804 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:50,811 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:50,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:50,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:50,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:50,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:50,812 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:50,812 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:50,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:50,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695010818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695010823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,912 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:50,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:50,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:50,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:50,918 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:50,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:50,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:50,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:50,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,920 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:50,920 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:50,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:50,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:50,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:50,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:50,921 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:50,921 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:50,921 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:50,922 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:50,922 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:50,922 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:50,922 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:50,922 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:50,923 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,923 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,924 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:50,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3677c57d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:50,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:50,925 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,928 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695010924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,929 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,929 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:50,929 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:50,929 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:50,929 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,929 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:50,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:50,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:50,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,932 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:50,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:50,934 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:50,935 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:50,935 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:50,935 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:50,935 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:50,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,936 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:50,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:50,937 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:50,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:50,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695010930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,939 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:50,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:50,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:50,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:50,942 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:50,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:50,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:50,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:50,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:50,942 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:50,943 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:50,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:50,943 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:50,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:50,943 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,043 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,046 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,049 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,050 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,051 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,051 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,051 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,051 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,051 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:51,052 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:51,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:51,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:51,052 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:51,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,055 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:51,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9bb69c1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:51,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:51,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,058 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:51,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:51,058 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:51,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,060 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:51,060 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:51,060 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:51,060 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:51,062 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,062 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,062 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,063 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,064 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:51,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,069 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,069 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:51,069 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,069 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695011131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695011141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,169 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,172 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,175 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,175 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,175 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,176 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,176 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,176 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:51,176 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:51,176 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:51,177 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:51,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:51,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@790e4ecd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:51,180 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:51,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,182 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:51,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:51,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:51,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:51,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:51,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:51,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:51,187 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:51,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:51,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:51,187 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:51,187 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:51,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,188 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,189 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695011187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695011187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:51,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:51,198 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,199 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:51,199 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:51,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/75c76ea87f834985be36540643676819 2023-03-24 21:55:51,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/6ca7e58bb3a344988135923483f05f97 2023-03-24 21:55:51,299 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,303 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,308 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,311 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,311 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,311 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:51,311 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:51,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:51,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0b2999[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:51,315 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:51,315 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:51,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:51,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:51,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:51,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:51,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:51,320 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:51,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:51,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:51,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:51,320 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:51,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,325 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,328 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:51,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:51,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:51,329 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:51,329 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:51,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,429 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,432 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,437 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,437 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,437 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:51,438 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:51,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:51,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:51,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:51,439 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:51,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17df5b28[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:51,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:51,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,443 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:51,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:51,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:51,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:51,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695011433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:51,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:51,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:51,448 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:51,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:51,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:51,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:51,448 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:51,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,450 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,452 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:51,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:51,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,456 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:51,456 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:51,455 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,456 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:51,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695011447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,556 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,571 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,574 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,575 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,587 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,587 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,587 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,587 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,587 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:51,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,591 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:51,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:51,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:51,592 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:51,594 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,594 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:51,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:51,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4578dd73[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:51,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:51,597 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:51,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:51,598 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:51,598 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,599 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:51,600 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,600 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:51,600 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,600 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:51,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:51,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:51,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:51,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:51,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:51,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:51,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,609 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:51,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:51,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:51,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:51,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:51,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:51,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:51,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:51,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:51,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:51,616 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:51,616 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:51,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:51,656 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e21ebea8c18e4fb581386918ce108b78 2023-03-24 21:55:51,662 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/75c76ea87f834985be36540643676819 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819 2023-03-24 21:55:51,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819, entries=200, sequenceid=374, filesize=14.3 K 2023-03-24 21:55:51,683 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/6ca7e58bb3a344988135923483f05f97 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97 2023-03-24 21:55:51,689 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97, entries=150, sequenceid=374, filesize=11.9 K 2023-03-24 21:55:51,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e21ebea8c18e4fb581386918ce108b78 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78 2023-03-24 21:55:51,699 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78, entries=150, sequenceid=374, filesize=11.9 K 2023-03-24 21:55:51,700 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 2a79b669e50783c6f283e5def38723af in 910ms, sequenceid=374, compaction requested=true 2023-03-24 21:55:51,700 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:51,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:51,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:51,700 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:51,700 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:51,700 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:51,701 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:51,701 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:51,701 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:51,701 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,701 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:51,701 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,701 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/01c2e2a98e314abcb9904f602c9131a2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=38.7 K 2023-03-24 21:55:51,701 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2974f7b2babe47069127f5e7c9985a9b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.3 K 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 01c2e2a98e314abcb9904f602c9131a2, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2974f7b2babe47069127f5e7c9985a9b, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8f604e62efc842ada15f0ebc4fde70fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679694948630 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e19d96fc381c4229837d6631b64cd486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679694948630 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 75c76ea87f834985be36540643676819, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:51,702 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6ca7e58bb3a344988135923483f05f97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:51,716 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:51,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:51,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:51,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:51,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:51,731 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:51,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:51,731 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:51,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,735 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,735 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:51,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:51,736 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:51,736 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:51,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:51,737 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#440 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:51,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:51,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:51,737 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:51,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:51,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:51,737 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:51,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:51,739 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:51,739 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:51,740 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:51,740 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:51,742 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#441 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:51,766 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/bc71785248aa4308ad5c69fd7a65b076 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/bc71785248aa4308ad5c69fd7a65b076 2023-03-24 21:55:51,775 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into bc71785248aa4308ad5c69fd7a65b076(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:51,775 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:51,775 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694951700; duration=0sec 2023-03-24 21:55:51,775 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:51,775 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:51,777 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:51,777 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:51,777 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:51,777 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b37851735c67425f9f66460c3f269a24, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.3 K 2023-03-24 21:55:51,777 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b37851735c67425f9f66460c3f269a24, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1679694948263 2023-03-24 21:55:51,778 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 260c8de726d0424f9d185d815932164d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=337, earliestPutTs=1679694948630 2023-03-24 21:55:51,778 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e21ebea8c18e4fb581386918ce108b78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:51,790 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#443 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:51,840 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/2bc7d11d6056451d9e106437196d2a3e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2bc7d11d6056451d9e106437196d2a3e 2023-03-24 21:55:51,846 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 2bc7d11d6056451d9e106437196d2a3e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:51,846 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:51,846 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694951700; duration=0sec 2023-03-24 21:55:51,846 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:51,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:51,952 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694952007, max timestamp=1679694952026Num uniques 859; 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694952007, max timestamp=1679694952026Num uniques 859; 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694952007, max timestamp=1679694952026Num uniques 859; 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:52,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:52,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695012109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695012121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,157 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d048869c194e4137ad2315e5500299f0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d048869c194e4137ad2315e5500299f0 2023-03-24 21:55:52,161 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into d048869c194e4137ad2315e5500299f0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:52,161 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:52,162 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694951700; duration=0sec 2023-03-24 21:55:52,162 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:52,185 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/5c80e2ad52d14558bd050accfcd1f764 2023-03-24 21:55:52,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695012195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,204 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/91647d6d52a44ddea304a12a6c2cc757 2023-03-24 21:55:52,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695012196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,228 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=384 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/850c85704f174d108b916db567bd98ef 2023-03-24 21:55:52,230 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695012221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,231 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/5c80e2ad52d14558bd050accfcd1f764 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764 2023-03-24 21:55:52,235 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764, entries=150, sequenceid=384, filesize=11.9 K 2023-03-24 21:55:52,236 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/91647d6d52a44ddea304a12a6c2cc757 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757 2023-03-24 21:55:52,240 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757, entries=150, sequenceid=384, filesize=11.9 K 2023-03-24 21:55:52,241 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/850c85704f174d108b916db567bd98ef as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef 2023-03-24 21:55:52,248 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef, entries=150, sequenceid=384, filesize=11.9 K 2023-03-24 21:55:52,249 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2a79b669e50783c6f283e5def38723af in 509ms, sequenceid=384, compaction requested=false 2023-03-24 21:55:52,249 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:52,249 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:52,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:52,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:52,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:52,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:52,251 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-03-24 21:55:52,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:52,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:52,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:52,251 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:52,251 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:52,252 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-24 21:55:52,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:52,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e8b9b37[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:52,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:52,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,258 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:52,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:52,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:52,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:52,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:52,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:52,260 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:52,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:52,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:52,263 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:52,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:52,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:52,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:52,263 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:52,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,266 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,267 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:52,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:52,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:52,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:52,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:52,273 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:52,273 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:52,319 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/5bf2912129d14039bc2700f08f0902a0 2023-03-24 21:55:52,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695012328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,371 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/15efc92cdf954bcdb8a8aa10e36f5ad7 2023-03-24 21:55:52,373 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:52,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:52,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:52,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:52,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:52,387 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:52,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:52,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:52,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:52,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,389 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:52,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:52,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:52,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:52,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:52,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:52,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:52,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:52,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:52,429 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:52,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695012433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:52,440 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:52,440 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:52,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:52,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,449 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695012440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,455 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:52,455 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:52,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=425 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/f6239f2ad40547819d48e498390415f8 2023-03-24 21:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,458 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:52,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:52,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@88a349e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:52,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:52,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,462 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,462 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:52,462 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:52,463 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:52,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/5bf2912129d14039bc2700f08f0902a0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0 2023-03-24 21:55:52,463 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,464 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:52,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:52,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:52,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:52,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0, entries=200, sequenceid=425, filesize=14.3 K 2023-03-24 21:55:52,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/15efc92cdf954bcdb8a8aa10e36f5ad7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7 2023-03-24 21:55:52,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:52,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:52,470 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:52,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:52,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:52,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:52,470 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:52,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:52,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:52,473 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:52,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:52,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,476 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:52,476 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:52,477 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:52,477 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:52,476 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:52,478 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:52,478 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:52,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:52,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:52,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:52,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7, entries=150, sequenceid=425, filesize=11.9 K 2023-03-24 21:55:52,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/f6239f2ad40547819d48e498390415f8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8 2023-03-24 21:55:52,494 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8, entries=150, sequenceid=425, filesize=11.9 K 2023-03-24 21:55:52,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~405.56 KB/415296, currentSize=40.25 KB/41220 for 2a79b669e50783c6f283e5def38723af in 244ms, sequenceid=425, compaction requested=true 2023-03-24 21:55:52,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:52,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:52,495 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:52,495 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:52,495 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:52,496 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:52,496 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39714 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:52,496 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:52,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:52,497 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:52,497 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d048869c194e4137ad2315e5500299f0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=38.8 K 2023-03-24 21:55:52,497 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,497 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/bc71785248aa4308ad5c69fd7a65b076, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.4 K 2023-03-24 21:55:52,497 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d048869c194e4137ad2315e5500299f0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:52,497 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bc71785248aa4308ad5c69fd7a65b076, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:52,498 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5c80e2ad52d14558bd050accfcd1f764, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679694950810 2023-03-24 21:55:52,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 91647d6d52a44ddea304a12a6c2cc757, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679694950810 2023-03-24 21:55:52,498 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5bf2912129d14039bc2700f08f0902a0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:52,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 15efc92cdf954bcdb8a8aa10e36f5ad7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:52,516 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:52,532 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:52,577 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:52,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:52,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:52,580 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/b9f8165d7d1d47e8a2d862e3ef5233fd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b9f8165d7d1d47e8a2d862e3ef5233fd 2023-03-24 21:55:52,587 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into b9f8165d7d1d47e8a2d862e3ef5233fd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:52,587 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:52,587 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694952495; duration=0sec 2023-03-24 21:55:52,587 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:52,587 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:52,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:52,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:52,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:52,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:52,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:52,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:52,588 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:52,588 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:52,588 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,589 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2bc7d11d6056451d9e106437196d2a3e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.4 K 2023-03-24 21:55:52,589 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2bc7d11d6056451d9e106437196d2a3e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1679694950043 2023-03-24 21:55:52,589 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 850c85704f174d108b916db567bd98ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=384, earliestPutTs=1679694950810 2023-03-24 21:55:52,589 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f6239f2ad40547819d48e498390415f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:52,590 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,590 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:52,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:52,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:52,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:52,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:52,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:52,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:52,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:52,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:52,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:52,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:52,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:52,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:52,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:52,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:52,595 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:52,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:52,611 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/3592872d9a3940cbbe9ea9d509d42bf6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/3592872d9a3940cbbe9ea9d509d42bf6 2023-03-24 21:55:52,612 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#451 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:52,624 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 3592872d9a3940cbbe9ea9d509d42bf6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:52,624 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:52,624 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694952495; duration=0sec 2023-03-24 21:55:52,624 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:52,656 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:52,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:52,672 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/26ca11f024bc4aee9633604ace55ed3d 2023-03-24 21:55:52,723 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/01fc86683c6645cbab8987dd43318f69 2023-03-24 21:55:52,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695012791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695012796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,805 DEBUG [Thread-3042] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4171 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:52,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695012802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695012903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:52,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:52,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695012913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,046 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e52927a31c394ec3893fc0134256905e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e52927a31c394ec3893fc0134256905e 2023-03-24 21:55:53,053 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into e52927a31c394ec3893fc0134256905e(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:53,054 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:53,054 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694952496; duration=0sec 2023-03-24 21:55:53,054 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:53,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695013113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695013123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,147 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/2004eaa48eed444da81c3db74f655f53 2023-03-24 21:55:53,153 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/26ca11f024bc4aee9633604ace55ed3d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d 2023-03-24 21:55:53,158 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d, entries=150, sequenceid=435, filesize=11.9 K 2023-03-24 21:55:53,161 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/01fc86683c6645cbab8987dd43318f69 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69 2023-03-24 21:55:53,175 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69, entries=150, sequenceid=435, filesize=11.9 K 2023-03-24 21:55:53,176 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/2004eaa48eed444da81c3db74f655f53 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53 2023-03-24 21:55:53,181 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53, entries=150, sequenceid=435, filesize=11.9 K 2023-03-24 21:55:53,182 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=134.18 KB/137400 for 2a79b669e50783c6f283e5def38723af in 587ms, sequenceid=435, compaction requested=false 2023-03-24 21:55:53,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:53,182 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:53,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:53,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:53,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:53,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:53,185 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,186 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,186 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:53,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:53,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65e128bc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:53,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:53,189 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,191 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:53,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:53,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:53,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:53,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:53,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:53,192 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:53,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:53,197 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:53,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:53,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:53,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:53,197 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:53,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:53,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,203 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,205 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:53,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:53,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:53,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,209 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,209 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:53,209 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:53,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:53,209 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:53,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:53,310 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:53,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:53,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:53,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:53,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:53,313 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:53,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:53,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:53,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:53,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:53,315 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:53,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:53,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:53,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:53,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:53,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:53,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:53,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:53,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:53,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:53,319 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=134.18 KB heapSize=399.89 KB 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:53,319 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:53,347 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/947bf4aa47ee441e9e37981c6bda5a55 2023-03-24 21:55:53,384 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/d341433e6c6848eb8d303916020a4b35 2023-03-24 21:55:53,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:53,426 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:53,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695013445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695013451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695013552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695013558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695013758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695013769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,808 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/3eb8988dd80c42e1831f1aaae3c25087 2023-03-24 21:55:53,811 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/947bf4aa47ee441e9e37981c6bda5a55 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55 2023-03-24 21:55:53,815 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55, entries=150, sequenceid=460, filesize=11.9 K 2023-03-24 21:55:53,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/d341433e6c6848eb8d303916020a4b35 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35 2023-03-24 21:55:53,818 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35, entries=150, sequenceid=460, filesize=11.9 K 2023-03-24 21:55:53,819 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/3eb8988dd80c42e1831f1aaae3c25087 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087 2023-03-24 21:55:53,822 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087, entries=150, sequenceid=460, filesize=11.9 K 2023-03-24 21:55:53,823 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~399.84 KB/409440, currentSize=40.25 KB/41220 for 2a79b669e50783c6f283e5def38723af in 504ms, sequenceid=460, compaction requested=true 2023-03-24 21:55:53,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:53,823 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:53,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:53,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:53,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:53,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:53,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,825 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,825 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:53,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6671f236[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:53,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:53,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,832 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:53,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:53,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:53,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:53,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:53,834 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:53,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:53,836 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:53,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:53,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:53,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:53,836 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:53,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,839 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:53,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:53,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:53,840 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:53,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:53,843 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:53,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:53,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:53,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:53,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:53,844 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:53,844 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:53,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:53,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:53,944 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:53,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:53,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:53,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:53,947 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:53,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:53,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:53,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:53,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,950 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:53,950 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:53,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:53,951 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:53,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:53,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:53,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:53,951 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:53,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:53,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:53,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:53,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:53,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:53,952 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:53,952 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:53,952 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:53,952 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-24 21:55:53,952 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:53,952 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:53,953 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:54,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:54,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694954153, max timestamp=1679694954176Num uniques 859; 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694954153, max timestamp=1679694954176Num uniques 859; 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694954153, max timestamp=1679694954176Num uniques 859; 2023-03-24 21:55:54,178 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:55:54,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695014226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695014227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695014233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,240 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695014234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695014335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695014335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695014342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695014342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,369 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/89b0c305b3044c4a87c993988924c05e 2023-03-24 21:55:54,400 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/d1d382ae3eb2410bacf03a08be33aa01 2023-03-24 21:55:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695014547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695014548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695014547, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:54,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695014549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,821 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=469 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ad83e88568a6465c92c55e263e41df99 2023-03-24 21:55:54,825 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/89b0c305b3044c4a87c993988924c05e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e 2023-03-24 21:55:54,829 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e, entries=150, sequenceid=469, filesize=11.9 K 2023-03-24 21:55:54,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/d1d382ae3eb2410bacf03a08be33aa01 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01 2023-03-24 21:55:54,842 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01, entries=150, sequenceid=469, filesize=11.9 K 2023-03-24 21:55:54,842 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/ad83e88568a6465c92c55e263e41df99 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99 2023-03-24 21:55:54,845 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99, entries=150, sequenceid=469, filesize=11.9 K 2023-03-24 21:55:54,846 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 2a79b669e50783c6f283e5def38723af in 894ms, sequenceid=469, compaction requested=true 2023-03-24 21:55:54,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:54,846 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:54,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:54,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:54,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:54,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:54,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,848 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:54,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74318fec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:54,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:54,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:54,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:54,852 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:54,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,852 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:54,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:54,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:54,853 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:54,853 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:54,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:54,857 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:54,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:54,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:54,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:54,857 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:54,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:54,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:54,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:54,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:54,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,863 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:54,864 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:54,864 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:54,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:54,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=154.31 KB heapSize=440.77 KB 2023-03-24 21:55:54,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:54,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7681, totalHeapSize=17712, min timestamp=1679694954224, max timestamp=1679694954865Num uniques 1068; 2023-03-24 21:55:54,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=48901, new segment data size=7681, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 21:55:54,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694954224, max timestamp=1679694954865Num uniques 1068; 2023-03-24 21:55:54,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:54,867 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7687, totalHeapSize=17712, min timestamp=1679694954224, max timestamp=1679694954865Num uniques 1068; 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:54,868 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=48907, new segment data size=7687, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:54,868 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:54,937 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694954892, max timestamp=1679694954916Num uniques 859; 2023-03-24 21:55:54,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:54,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694954892, max timestamp=1679694954928Num uniques 859; 2023-03-24 21:55:54,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694954892, max timestamp=1679694954912Num uniques 859; 2023-03-24 21:55:54,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-24 21:55:54,938 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:55:54,964 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:54,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:54,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:54,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:54,968 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:54,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:54,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:54,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:54,968 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,969 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:54,971 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:54,971 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:54,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,972 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:54,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:54,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:54,973 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:54,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:54,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:54,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:54,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:54,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:54,980 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:54,980 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:54,980 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:54,980 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:54,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:54,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,982 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:54,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:54,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d466b30[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:54,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:54,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:54,985 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:54,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:54,986 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,986 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:54,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:54,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:54,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:54,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:54,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:54,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:54,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:54,990 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:54,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,995 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:54,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:54,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,000 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,001 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,001 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,001 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:55,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694954991, max timestamp=1679694955009Num uniques 1018; 2023-03-24 21:55:55,011 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 21:55:55,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694954991, max timestamp=1679694955009Num uniques 1018; 2023-03-24 21:55:55,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-03-24 21:55:55,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1679694954991, max timestamp=1679694955009Num uniques 1009; 2023-03-24 21:55:55,012 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=9978, suffix heap size=132904,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=218 2023-03-24 21:55:55,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695015049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694955027, max timestamp=1679694955062Num uniques 1077; 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694955027, max timestamp=1679694955062Num uniques 1077; 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1679694955027, max timestamp=1679694955062Num uniques 1077; 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:55:55,065 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=12682, suffix heap size=139976,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=277 2023-03-24 21:55:55,101 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,104 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,105 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,106 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,107 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,108 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,110 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@418e02ec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,122 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,122 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,126 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,130 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,130 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,130 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:55,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695015125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695015127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695015128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695015166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,230 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,240 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,242 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,242 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,247 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,247 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,247 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,247 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,250 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,251 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2887a8a9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,254 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48196 deadline: 1679695015242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695015245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,257 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695015245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,258 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,258 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,261 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,263 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,267 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,267 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,268 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,268 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/daaae95fa5a24f34adc90f95482607af 2023-03-24 21:55:55,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/aae792c8beb1402383c1a680d6c7a107 2023-03-24 21:55:55,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=509 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/6e4c287771b44f61b9541b403279e6dc 2023-03-24 21:55:55,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/daaae95fa5a24f34adc90f95482607af as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af 2023-03-24 21:55:55,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af, entries=200, sequenceid=509, filesize=14.3 K 2023-03-24 21:55:55,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/aae792c8beb1402383c1a680d6c7a107 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107 2023-03-24 21:55:55,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107, entries=150, sequenceid=509, filesize=11.9 K 2023-03-24 21:55:55,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/6e4c287771b44f61b9541b403279e6dc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc 2023-03-24 21:55:55,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc, entries=150, sequenceid=509, filesize=11.9 K 2023-03-24 21:55:55,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~105.19 KB/107712, currentSize=147.60 KB/151140 for 2a79b669e50783c6f283e5def38723af in 481ms, sequenceid=509, compaction requested=true 2023-03-24 21:55:55,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:55,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:55,347 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:55,347 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:55,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:55,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:55,348 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64188 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:55,348 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:55,349 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:55,349 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:55,349 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,349 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,349 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b9f8165d7d1d47e8a2d862e3ef5233fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=62.7 K 2023-03-24 21:55:55,349 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/3592872d9a3940cbbe9ea9d509d42bf6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=60.3 K 2023-03-24 21:55:55,349 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting b9f8165d7d1d47e8a2d862e3ef5233fd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:55,349 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3592872d9a3940cbbe9ea9d509d42bf6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:55,349 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 26ca11f024bc4aee9633604ace55ed3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1679694952267 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 01fc86683c6645cbab8987dd43318f69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1679694952267 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 947bf4aa47ee441e9e37981c6bda5a55, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679694952778 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d341433e6c6848eb8d303916020a4b35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679694952778 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 89b0c305b3044c4a87c993988924c05e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679694953430 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d1d382ae3eb2410bacf03a08be33aa01, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679694953430 2023-03-24 21:55:55,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting daaae95fa5a24f34adc90f95482607af, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954233 2023-03-24 21:55:55,351 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting aae792c8beb1402383c1a680d6c7a107, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954864 2023-03-24 21:55:55,363 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:55,368 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,369 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#465 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:55,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,379 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,381 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,382 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:55,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955123, max timestamp=1679694955384Num uniques 1136; 2023-03-24 21:55:55,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 21:55:55,387 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=111.82 KB heapSize=310.83 KB 2023-03-24 21:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,387 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:55,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955123, max timestamp=1679694955384Num uniques 1136; 2023-03-24 21:55:55,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 21:55:55,387 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955123, max timestamp=1679694955384Num uniques 1136; 2023-03-24 21:55:55,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-03-24 21:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:55,389 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,391 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,391 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,396 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/72bd9d81e612489b8ae3c88340e9432a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/72bd9d81e612489b8ae3c88340e9432a 2023-03-24 21:55:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,403 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,403 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,406 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/7d73818d72a74b69bcdc5ef0689a715d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/7d73818d72a74b69bcdc5ef0689a715d 2023-03-24 21:55:55,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,412 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 7d73818d72a74b69bcdc5ef0689a715d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:55,412 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 72bd9d81e612489b8ae3c88340e9432a(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:55,412 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:55,412 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:55,412 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=11, startTime=1679694955347; duration=0sec 2023-03-24 21:55:55,412 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=11, startTime=1679694955347; duration=0sec 2023-03-24 21:55:55,412 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:55,412 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:55,412 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-24 21:55:55,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,414 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61748 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-24 21:55:55,414 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:55,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,414 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,414 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,415 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e52927a31c394ec3893fc0134256905e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=60.3 K 2023-03-24 21:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,415 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e52927a31c394ec3893fc0134256905e, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=425, earliestPutTs=1679694952058 2023-03-24 21:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,415 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2004eaa48eed444da81c3db74f655f53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1679694952267 2023-03-24 21:55:55,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,416 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 3eb8988dd80c42e1831f1aaae3c25087, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679694952778 2023-03-24 21:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50311cba[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,416 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ad83e88568a6465c92c55e263e41df99, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=469, earliestPutTs=1679694953430 2023-03-24 21:55:55,416 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6e4c287771b44f61b9541b403279e6dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954864 2023-03-24 21:55:55,417 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,419 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,419 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,419 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,419 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/025d5c096aa340b6abd3ff2acd5a8e09 2023-03-24 21:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,427 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#467 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,430 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,433 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,433 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2a3964896f214a76a6ade891b20d37a3 2023-03-24 21:55:55,448 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/fd208bcd70ea49b7a49d0f751e15a173 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/fd208bcd70ea49b7a49d0f751e15a173 2023-03-24 21:55:55,455 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into fd208bcd70ea49b7a49d0f751e15a173(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:55,455 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:55,455 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=11, startTime=1679694955347; duration=0sec 2023-03-24 21:55:55,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:55,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955456, max timestamp=1679694955477Num uniques 859; 2023-03-24 21:55:55,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955456, max timestamp=1679694955477Num uniques 859; 2023-03-24 21:55:55,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955456, max timestamp=1679694955477Num uniques 859; 2023-03-24 21:55:55,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,480 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694955485, max timestamp=1679694955513Num uniques 1118; 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694955485, max timestamp=1679694955513Num uniques 1118; 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679694955485, max timestamp=1679694955513Num uniques 1118; 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:55,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=12268, suffix heap size=144240,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=268 2023-03-24 21:55:55,534 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,543 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,543 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,546 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,546 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,546 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679694955528, max timestamp=1679694955546Num uniques 1127; 2023-03-24 21:55:55,548 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-24 21:55:55,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679694955528, max timestamp=1679694955546Num uniques 1127; 2023-03-24 21:55:55,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-24 21:55:55,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=327, cellSize=14972, totalHeapSize=34248, min timestamp=1679694955528, max timestamp=1679694955546Num uniques 1127; 2023-03-24 21:55:55,549 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=14972, suffix heap size=145176,new segment heap size=34248 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=327 2023-03-24 21:55:55,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,551 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,551 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,553 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e30e267[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,557 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,557 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,557 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,582 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,582 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695015595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:55,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:48234 deadline: 1679695015595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955581, max timestamp=1679694955604Num uniques 1186; 2023-03-24 21:55:55,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-24 21:55:55,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955581, max timestamp=1679694955604Num uniques 1186; 2023-03-24 21:55:55,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-24 21:55:55,608 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679694955581, max timestamp=1679694955604Num uniques 1186; 2023-03-24 21:55:55,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=54316, new segment data size=8516, suffix heap size=151312,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1186, new segment cells count=186 2023-03-24 21:55:55,669 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694955651, max timestamp=1679694955667Num uniques 1045; 2023-03-24 21:55:55,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:55,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694955651, max timestamp=1679694955667Num uniques 1045; 2023-03-24 21:55:55,670 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:55,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1679694955651, max timestamp=1679694955667Num uniques 1045; 2023-03-24 21:55:55,671 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-03-24 21:55:55,682 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,686 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,690 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,702 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,702 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f98102f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,707 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,707 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,710 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,712 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,712 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,715 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,718 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:55,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,722 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,722 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694955709, max timestamp=1679694955725Num uniques 1104; 2023-03-24 21:55:55,728 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:55:55,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694955709, max timestamp=1679694955725Num uniques 1104; 2023-03-24 21:55:55,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694955709, max timestamp=1679694955725Num uniques 1104; 2023-03-24 21:55:55,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:55:55,730 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679694955750, max timestamp=1679694955761Num uniques 1013; 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679694955750, max timestamp=1679694955761Num uniques 1013; 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679694955750, max timestamp=1679694955761Num uniques 1013; 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:55:55,763 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:55:55,822 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,825 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,829 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,829 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,829 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,829 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,829 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,830 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679694955781, max timestamp=1679694955809Num uniques 1022; 2023-03-24 21:55:55,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-24 21:55:55,830 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679694955781, max timestamp=1679694955809Num uniques 1022; 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=237, cellSize=10852, totalHeapSize=24888, min timestamp=1679694955781, max timestamp=1679694955809Num uniques 1022; 2023-03-24 21:55:55,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=10852, suffix heap size=134256,new segment heap size=24888 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=237 2023-03-24 21:55:55,832 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,833 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,833 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6acf3d95[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,841 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,850 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,850 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,852 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,853 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:55,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:55,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,857 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:55,858 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:55,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=581 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/43abee2cfc9544fd95cdb666d5c278f8 2023-03-24 21:55:55,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/025d5c096aa340b6abd3ff2acd5a8e09 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09 2023-03-24 21:55:55,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09, entries=200, sequenceid=581, filesize=14.3 K 2023-03-24 21:55:55,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2a3964896f214a76a6ade891b20d37a3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3 2023-03-24 21:55:55,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1679694955846, max timestamp=1679694955875Num uniques 1131; 2023-03-24 21:55:55,877 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=12866, suffix heap size=145592,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=281 2023-03-24 21:55:55,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694955846, max timestamp=1679694955875Num uniques 1131; 2023-03-24 21:55:55,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-24 21:55:55,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694955846, max timestamp=1679694955875Num uniques 1096; 2023-03-24 21:55:55,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=50196, new segment data size=10576, suffix heap size=141952,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1096, new segment cells count=231 2023-03-24 21:55:55,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3, entries=150, sequenceid=581, filesize=11.9 K 2023-03-24 21:55:55,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/43abee2cfc9544fd95cdb666d5c278f8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8 2023-03-24 21:55:55,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8, entries=150, sequenceid=581, filesize=11.9 K 2023-03-24 21:55:55,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~63.66 KB/65184, currentSize=55.91 KB/57250 for 2a79b669e50783c6f283e5def38723af in 502ms, sequenceid=581, compaction requested=false 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:55,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:55,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=62.62 KB heapSize=164.17 KB 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:55,889 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:55,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/45348c929f6b445eb45d9b91db362080 2023-03-24 21:55:55,944 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/4c2ca94b54e54f81acd17a7e165031d4 2023-03-24 21:55:55,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955924, max timestamp=1679694955953Num uniques 859; 2023-03-24 21:55:55,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955924, max timestamp=1679694955953Num uniques 859; 2023-03-24 21:55:55,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,956 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694955924, max timestamp=1679694955953Num uniques 859; 2023-03-24 21:55:55,958 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:55,958 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:55,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:55,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:55,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:55,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:55,975 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:55,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:55,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:55,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:55,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:55,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:55,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:55,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:55,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:55,987 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:55,987 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:55,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:55,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:55,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:55,988 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:55,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:55,989 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,989 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,990 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:55,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c338ba[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:55,991 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:55,991 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:55,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:55,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:55,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:55,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:55,993 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,993 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:55,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:55,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:55,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:55,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:55,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:55,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:55,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:55,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:55,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:55,999 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:55,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,004 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,004 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695016028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695016029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695016030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695016038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,104 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,108 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,110 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,112 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,112 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,112 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:56,112 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,115 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b29c75f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,118 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,118 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,120 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,120 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,122 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,122 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,124 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,125 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,128 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,131 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:56,132 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,132 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695016139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695016143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,154 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695016144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695016148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,233 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,236 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,237 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,240 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:56,242 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,245 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@448fa97d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,247 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,247 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,252 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,258 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,258 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,258 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,265 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,265 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:56,274 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,275 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,275 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,349 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695016344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695016356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695016356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,364 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695016358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,375 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,379 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,381 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,381 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,382 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,383 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,383 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,383 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:56,383 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,384 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,386 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=746 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/c7d629f2c7b84239a72db248b840d8ac 2023-03-24 21:55:56,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@573f17ec[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,387 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,390 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,390 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,391 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/45348c929f6b445eb45d9b91db362080 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080 2023-03-24 21:55:56,392 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080, entries=200, sequenceid=746, filesize=14.3 K 2023-03-24 21:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,396 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,396 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,396 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,396 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/4c2ca94b54e54f81acd17a7e165031d4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4 2023-03-24 21:55:56,397 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,397 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,397 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,399 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4, entries=150, sequenceid=746, filesize=11.9 K 2023-03-24 21:55:56,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/c7d629f2c7b84239a72db248b840d8ac as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac 2023-03-24 21:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,405 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,405 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,405 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac, entries=150, sequenceid=746, filesize=11.9 K 2023-03-24 21:55:56,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~62.62 KB/64120, heapSize ~164.13 KB/168064, currentSize=127.47 KB/130530 for 2a79b669e50783c6f283e5def38723af in 517ms, sequenceid=746, compaction requested=true 2023-03-24 21:55:56,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:56,407 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:56,407 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:56,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:56,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:56,408 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42426 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:56,408 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:56,408 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:56,408 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,408 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:56,408 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/7d73818d72a74b69bcdc5ef0689a715d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=41.4 K 2023-03-24 21:55:56,408 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,408 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/72bd9d81e612489b8ae3c88340e9432a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.7 K 2023-03-24 21:55:56,409 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7d73818d72a74b69bcdc5ef0689a715d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954864 2023-03-24 21:55:56,410 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 72bd9d81e612489b8ae3c88340e9432a, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954864 2023-03-24 21:55:56,410 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 025d5c096aa340b6abd3ff2acd5a8e09, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1679694955123 2023-03-24 21:55:56,410 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2a3964896f214a76a6ade891b20d37a3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1679694955124 2023-03-24 21:55:56,411 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 45348c929f6b445eb45d9b91db362080, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:56,411 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4c2ca94b54e54f81acd17a7e165031d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:56,420 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#473 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:56,424 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:56,445 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/5a8d671e078a4be7b1031a7b899b7e4b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5a8d671e078a4be7b1031a7b899b7e4b 2023-03-24 21:55:56,447 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/d2fd8294ba7b46b5930b545b47cb5d8c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d2fd8294ba7b46b5930b545b47cb5d8c 2023-03-24 21:55:56,452 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 5a8d671e078a4be7b1031a7b899b7e4b(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:56,452 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,452 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694956407; duration=0sec 2023-03-24 21:55:56,453 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:56,453 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:56,454 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into d2fd8294ba7b46b5930b545b47cb5d8c(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:56,454 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,454 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694956407; duration=0sec 2023-03-24 21:55:56,455 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:56,455 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:56,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:56,456 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,456 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/fd208bcd70ea49b7a49d0f751e15a173, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.7 K 2023-03-24 21:55:56,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting fd208bcd70ea49b7a49d0f751e15a173, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=509, earliestPutTs=1679694954864 2023-03-24 21:55:56,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 43abee2cfc9544fd95cdb666d5c278f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=581, earliestPutTs=1679694955124 2023-03-24 21:55:56,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c7d629f2c7b84239a72db248b840d8ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:56,461 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:56,505 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,511 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,511 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,534 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,534 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,535 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:56,535 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,549 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/35954058edda492dad94d515d96cb758 2023-03-24 21:55:56,569 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5a04d8b2749a484c9b1c9928c8f35196 2023-03-24 21:55:56,585 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=782 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/4c0a6d3699eb4afc9c26bb263ae422c8 2023-03-24 21:55:56,591 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/35954058edda492dad94d515d96cb758 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758 2023-03-24 21:55:56,596 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758, entries=150, sequenceid=782, filesize=11.9 K 2023-03-24 21:55:56,596 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5a04d8b2749a484c9b1c9928c8f35196 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196 2023-03-24 21:55:56,607 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196, entries=150, sequenceid=782, filesize=11.9 K 2023-03-24 21:55:56,608 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/4c0a6d3699eb4afc9c26bb263ae422c8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8 2023-03-24 21:55:56,611 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8, entries=150, sequenceid=782, filesize=11.9 K 2023-03-24 21:55:56,612 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=0 B/0 for 2a79b669e50783c6f283e5def38723af in 77ms, sequenceid=782, compaction requested=false 2023-03-24 21:55:56,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,615 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,616 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,616 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,616 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,616 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,618 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c12e359[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,618 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,619 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,620 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,619 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,620 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,623 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,626 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,626 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,630 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,630 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,630 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,639 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,639 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,640 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,640 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,640 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:56,670 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:56,670 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:56,684 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/bc4d7e865b5e4a5a9570fca5b4a158bc 2023-03-24 21:55:56,696 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5424f4549b90455eb15f8273064a5928 2023-03-24 21:55:56,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e2fdfe09a8024ce490a9c1d376748ac9 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694956694, max timestamp=1679694956707Num uniques 859; 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694956694, max timestamp=1679694956707Num uniques 859; 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694956694, max timestamp=1679694956707Num uniques 859; 2023-03-24 21:55:56,709 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:55:56,712 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/bc4d7e865b5e4a5a9570fca5b4a158bc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc 2023-03-24 21:55:56,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc, entries=100, sequenceid=792, filesize=9.5 K 2023-03-24 21:55:56,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5424f4549b90455eb15f8273064a5928 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928 2023-03-24 21:55:56,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928, entries=100, sequenceid=792, filesize=9.5 K 2023-03-24 21:55:56,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/e2fdfe09a8024ce490a9c1d376748ac9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9 2023-03-24 21:55:56,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9, entries=100, sequenceid=792, filesize=9.5 K 2023-03-24 21:55:56,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=73.80 KB/75570 for 2a79b669e50783c6f283e5def38723af in 54ms, sequenceid=792, compaction requested=true 2023-03-24 21:55:56,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:56,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:55:56,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-24 21:55:56,724 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-24 21:55:56,725 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 21:55:56,725 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 21:55:56,725 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. because compaction request was cancelled 2023-03-24 21:55:56,725 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:56,726 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:56,726 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:56,726 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,726 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5a8d671e078a4be7b1031a7b899b7e4b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.4 K 2023-03-24 21:55:56,726 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5a8d671e078a4be7b1031a7b899b7e4b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:56,727 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 35954058edda492dad94d515d96cb758, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1679694955986 2023-03-24 21:55:56,727 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting bc4d7e865b5e4a5a9570fca5b4a158bc, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694956659 2023-03-24 21:55:56,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:56,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=80.51 KB heapSize=216.23 KB 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:56,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:55:56,736 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:56,743 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,751 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,759 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,759 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,759 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:56,759 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,759 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,760 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,762 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,764 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49b4d0bb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,765 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,765 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,766 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,767 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,767 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,768 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,770 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,770 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,773 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,787 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,787 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695016777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695016778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695016779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695016792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48210 deadline: 1679695016819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,832 DEBUG [Thread-3042] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8199 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:55:56,881 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/7dc7b291e1e8403eaf33feece10122c8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7dc7b291e1e8403eaf33feece10122c8 2023-03-24 21:55:56,885 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 7dc7b291e1e8403eaf33feece10122c8(size=13.0 K), total size for store is 34.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:56,885 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:56,885 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694956407; duration=0sec 2023-03-24 21:55:56,885 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:56,885 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:56,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35204 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:56,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:56,886 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,886 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d2fd8294ba7b46b5930b545b47cb5d8c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=34.4 K 2023-03-24 21:55:56,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d2fd8294ba7b46b5930b545b47cb5d8c, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:56,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5a04d8b2749a484c9b1c9928c8f35196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1679694955986 2023-03-24 21:55:56,887 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5424f4549b90455eb15f8273064a5928, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694956659 2023-03-24 21:55:56,887 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:56,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:56,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:56,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:56,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:56,891 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:56,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:56,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:56,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,893 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,893 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:56,894 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,894 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:56,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:56,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:56,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:56,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:56,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:56,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:56,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:56,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:56,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:56,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:56,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:56,900 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:56,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:56,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:56,901 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:56,903 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:56,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695016894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695016894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,904 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695016894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,904 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,905 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:56,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70495cb1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:56,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:56,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:56,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695016902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,909 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:56,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:56,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:56,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:56,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:56,914 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,914 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:56,919 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:56,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:56,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:56,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:56,919 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:56,921 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,921 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:56,921 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,921 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,923 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:56,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:56,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:56,925 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,926 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:56,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:56,926 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:56,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:56,927 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:56,927 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:56,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:56,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:56,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:56,927 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:56,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,027 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,030 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,030 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,032 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,038 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,038 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,038 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,039 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:57,039 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:57,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,045 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,045 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:57,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe6a523[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:57,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:57,047 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:57,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:57,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:57,048 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:57,050 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:57,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,050 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:57,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:57,053 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:57,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:57,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:57,053 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:57,054 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:57,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,055 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,056 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,057 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:57,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:57,061 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,061 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,062 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:57,062 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695017105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695017105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695017105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695017109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/6cdd44853148478f8500d6ce0c24f267 2023-03-24 21:55:57,160 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/63354df29448463d88fee01ef507477b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/63354df29448463d88fee01ef507477b 2023-03-24 21:55:57,162 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,164 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 63354df29448463d88fee01ef507477b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:57,164 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:57,165 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=13, startTime=1679694956724; duration=0sec 2023-03-24 21:55:57,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,165 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:57,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,166 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,169 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:57,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,170 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,180 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,180 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,180 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,180 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:57,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:57,182 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,182 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,182 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5f3e6c11dddc46128a80825f4ca019b4 2023-03-24 21:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,183 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:57,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34858dad[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:57,185 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:57,185 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:57,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:57,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:57,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:57,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:57,191 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,191 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,192 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:57,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:57,195 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:57,195 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:57,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:57,196 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:57,196 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:57,197 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,197 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,197 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,198 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,199 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:57,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,202 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:57,202 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:57,302 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,305 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,305 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:57,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:57,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:57,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,315 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2440fae244714b159ff397eee5c1299d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2440fae244714b159ff397eee5c1299d 2023-03-24 21:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:57,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b5bb28e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:57,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:57,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:57,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:57,318 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:57,319 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into 2440fae244714b159ff397eee5c1299d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:57,319 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:57,319 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=13, startTime=1679694956724; duration=0sec 2023-03-24 21:55:57,319 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,320 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:57,320 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:57,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:57,320 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:57,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,323 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,325 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,328 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:57,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,328 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,328 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:57,329 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:57,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,329 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695017418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695017419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695017419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,429 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695017421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,432 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,434 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,435 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:57,435 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,435 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,435 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,435 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,435 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:57,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:57,437 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61cdd180[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:57,440 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:57,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:57,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:57,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:57,448 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:57,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,448 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:57,451 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:57,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:57,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:57,451 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:57,451 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:57,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,458 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,458 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:57,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:57,460 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:57,460 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:57,560 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,565 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,566 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,568 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,571 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,571 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,572 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:57,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:57,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:57,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:57,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,574 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:57,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b68bf27[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:57,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:57,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:57,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:57,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:57,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:57,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:57,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:57,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:57,582 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:57,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:57,582 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:57,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,585 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,586 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,586 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:57,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:57,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:57,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:57,591 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:57,591 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:57,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:57,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:57,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:57,617 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/da66716104fa4954a9bc32cce5ab1609 2023-03-24 21:55:57,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/6cdd44853148478f8500d6ce0c24f267 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267 2023-03-24 21:55:57,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267, entries=250, sequenceid=820, filesize=16.7 K 2023-03-24 21:55:57,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/5f3e6c11dddc46128a80825f4ca019b4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4 2023-03-24 21:55:57,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4, entries=150, sequenceid=820, filesize=11.9 K 2023-03-24 21:55:57,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/da66716104fa4954a9bc32cce5ab1609 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609 2023-03-24 21:55:57,634 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609, entries=150, sequenceid=820, filesize=11.9 K 2023-03-24 21:55:57,635 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~236.11 KB/241776, currentSize=93.93 KB/96180 for 2a79b669e50783c6f283e5def38723af in 907ms, sequenceid=820, compaction requested=true 2023-03-24 21:55:57,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:57,635 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 21:55:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:57,635 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:55:57,635 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:57,636 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 21:55:57,636 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 21:55:57,636 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. because compaction request was cancelled 2023-03-24 21:55:57,636 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-24 21:55:57,637 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47390 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:57,637 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 21:55:57,637 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 21:55:57,637 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:57,637 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. because compaction request was cancelled 2023-03-24 21:55:57,637 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,637 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7dc7b291e1e8403eaf33feece10122c8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=46.3 K 2023-03-24 21:55:57,638 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7dc7b291e1e8403eaf33feece10122c8, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=746, earliestPutTs=1679694955876 2023-03-24 21:55:57,638 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4c0a6d3699eb4afc9c26bb263ae422c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=782, earliestPutTs=1679694955986 2023-03-24 21:55:57,638 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e2fdfe09a8024ce490a9c1d376748ac9, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694956659 2023-03-24 21:55:57,639 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting da66716104fa4954a9bc32cce5ab1609, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694956717 2023-03-24 21:55:57,645 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#487 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:57,692 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:57,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:57,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:57,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:57,695 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:57,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:57,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:57,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:57,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:57,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:57,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:57,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:57,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:57,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:57,699 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:57,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:57,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:57,930 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:57,959 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695017951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695017952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,962 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695017953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:57,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:57,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695017959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,059 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/7229e71d55b94262b07c19a6b3dff296 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7229e71d55b94262b07c19a6b3dff296 2023-03-24 21:55:58,064 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into 7229e71d55b94262b07c19a6b3dff296(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:58,065 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,065 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=12, startTime=1679694957635; duration=0sec 2023-03-24 21:55:58,065 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:58,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695018060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695018063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695018069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695018069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,115 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e043a50b88c44ef68c14e5268dd4b601 2023-03-24 21:55:58,125 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/84ab6131692c43a2a075a66675507cf5 2023-03-24 21:55:58,135 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=840 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/00292e388841411fb538caf0529572f5 2023-03-24 21:55:58,140 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e043a50b88c44ef68c14e5268dd4b601 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601 2023-03-24 21:55:58,143 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 21:55:58,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/84ab6131692c43a2a075a66675507cf5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5 2023-03-24 21:55:58,147 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 21:55:58,147 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/00292e388841411fb538caf0529572f5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5 2023-03-24 21:55:58,151 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5, entries=150, sequenceid=840, filesize=11.9 K 2023-03-24 21:55:58,152 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2a79b669e50783c6f283e5def38723af in 453ms, sequenceid=840, compaction requested=true 2023-03-24 21:55:58,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1398a302[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,158 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,158 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,158 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,158 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:58,165 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,165 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,165 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,169 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,177 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,181 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,181 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,181 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:58,274 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:58,274 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,281 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:58,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:58,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:58,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:58,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:58,286 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:58,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:58,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:58,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,289 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:58,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:58,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:58,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:58,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:58,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:58,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:58,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:58,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:58,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:58,291 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:58,291 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/c73cfce0e235492f9aa34cdd357e0440 2023-03-24 21:55:58,294 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,294 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1551bc3d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,303 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,304 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2e201cb4516543ba96ae46c6ddeaf673 2023-03-24 21:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:58,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,309 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,309 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,309 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695018295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,315 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,318 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695018306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695018307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695018309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,320 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,320 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:58,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,331 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=857 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/23bd1d2229a1480fa6717d65f295bcfb 2023-03-24 21:55:58,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/c73cfce0e235492f9aa34cdd357e0440 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440 2023-03-24 21:55:58,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440, entries=200, sequenceid=857, filesize=14.3 K 2023-03-24 21:55:58,338 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/2e201cb4516543ba96ae46c6ddeaf673 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673 2023-03-24 21:55:58,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673, entries=150, sequenceid=857, filesize=11.9 K 2023-03-24 21:55:58,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/23bd1d2229a1480fa6717d65f295bcfb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb 2023-03-24 21:55:58,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb, entries=150, sequenceid=857, filesize=11.9 K 2023-03-24 21:55:58,346 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 2a79b669e50783c6f283e5def38723af in 72ms, sequenceid=857, compaction requested=true 2023-03-24 21:55:58,346 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:58,346 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:58,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:58,346 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:55:58,346 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:58,347 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 57256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/A is initiating minor compaction (all files) 2023-03-24 21:55:58,348 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/A in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,348 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/63354df29448463d88fee01ef507477b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=55.9 K 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/B is initiating minor compaction (all files) 2023-03-24 21:55:58,348 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/B in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 63354df29448463d88fee01ef507477b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694955986 2023-03-24 21:55:58,348 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2440fae244714b159ff397eee5c1299d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=48.8 K 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6cdd44853148478f8500d6ce0c24f267, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694956707 2023-03-24 21:55:58,348 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2440fae244714b159ff397eee5c1299d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1679694955986 2023-03-24 21:55:58,349 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e043a50b88c44ef68c14e5268dd4b601, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679694956777 2023-03-24 21:55:58,349 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5f3e6c11dddc46128a80825f4ca019b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694956717 2023-03-24 21:55:58,349 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c73cfce0e235492f9aa34cdd357e0440, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1679694957949 2023-03-24 21:55:58,350 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 84ab6131692c43a2a075a66675507cf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679694956777 2023-03-24 21:55:58,350 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2e201cb4516543ba96ae46c6ddeaf673, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1679694957949 2023-03-24 21:55:58,364 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#A#compaction#494 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:58,365 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#B#compaction#495 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:58,384 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/0ae30b6c928d44e282a6bdb7c2ec7a33 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/0ae30b6c928d44e282a6bdb7c2ec7a33 2023-03-24 21:55:58,386 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/fa9c3488bf90422dbe3762bad90941da as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fa9c3488bf90422dbe3762bad90941da 2023-03-24 21:55:58,389 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/A of 2a79b669e50783c6f283e5def38723af into 0ae30b6c928d44e282a6bdb7c2ec7a33(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:58,389 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,389 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/A, priority=12, startTime=1679694958346; duration=0sec 2023-03-24 21:55:58,389 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:55:58,390 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:55:58,392 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:55:58,392 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 2a79b669e50783c6f283e5def38723af/C is initiating minor compaction (all files) 2023-03-24 21:55:58,392 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2a79b669e50783c6f283e5def38723af/C in TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,393 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7229e71d55b94262b07c19a6b3dff296, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp, totalSize=36.9 K 2023-03-24 21:55:58,393 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7229e71d55b94262b07c19a6b3dff296, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694956717 2023-03-24 21:55:58,393 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 00292e388841411fb538caf0529572f5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=840, earliestPutTs=1679694956777 2023-03-24 21:55:58,393 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 23bd1d2229a1480fa6717d65f295bcfb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=857, earliestPutTs=1679694957949 2023-03-24 21:55:58,404 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2a79b669e50783c6f283e5def38723af#C#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:55:58,406 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 2a79b669e50783c6f283e5def38723af/B of 2a79b669e50783c6f283e5def38723af into fa9c3488bf90422dbe3762bad90941da(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:58,406 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,406 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/B, priority=12, startTime=1679694958346; duration=0sec 2023-03-24 21:55:58,406 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:58,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 2a79b669e50783c6f283e5def38723af 2023-03-24 21:55:58,417 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:58,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:58,420 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:58,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:58,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:58,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:58,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:58,433 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:58,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:58,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:58,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:58,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:58,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:58,441 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:58,441 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:58,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:58,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:58,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a9f3ec6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,448 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695018442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695018443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,453 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695018444, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,454 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,456 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695018449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,456 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,457 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:58,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,462 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,462 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,462 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,467 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,475 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,480 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,480 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,481 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,480 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,556 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695018549, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,559 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695018554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695018557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695018563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,581 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:58,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:58,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:58,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:58,584 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:58,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:58,584 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:58,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:58,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:58,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:58,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:58,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:58,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,588 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:58,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:58,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:58,595 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,595 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,597 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,598 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54c02b35[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,600 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,601 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,601 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,603 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,603 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,604 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,604 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,604 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,604 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,604 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,604 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,606 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,607 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,607 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,611 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,613 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,613 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,614 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,714 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:58,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:58,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:58,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:58,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:58,717 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:58,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:58,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:58,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,719 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:58,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:58,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:58,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:58,722 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:58,722 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,722 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:58,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:58,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:58,727 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,727 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,728 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,729 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@216021c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,732 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,734 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,734 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,734 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,735 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:58,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,740 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,740 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,740 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,740 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,740 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,741 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,742 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,749 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,750 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,750 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,750 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,750 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:58,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695018758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695018760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695018761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:58,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695018771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,841 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/f03e48a5daf640f98aea705421be4add as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f03e48a5daf640f98aea705421be4add 2023-03-24 21:55:58,845 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2a79b669e50783c6f283e5def38723af/C of 2a79b669e50783c6f283e5def38723af into f03e48a5daf640f98aea705421be4add(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:55:58,846 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:58,846 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af., storeName=2a79b669e50783c6f283e5def38723af/C, priority=13, startTime=1679694958346; duration=0sec 2023-03-24 21:55:58,846 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:55:58,850 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:58,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:58,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:58,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:58,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:58,854 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:58,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:58,854 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:58,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,857 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,858 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:58,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:58,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:58,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:58,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:58,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:58,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:58,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:58,862 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,862 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:58,863 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:58,863 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:58,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:58,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:58,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:58,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:58,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,865 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:58,867 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:58,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1abef637[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:58,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:58,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,872 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e083da2215804cf1980a9db2ad9d78f7 2023-03-24 21:55:58,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:58,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:58,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:58,872 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:58,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:58,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:58,874 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:58,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:58,877 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:58,877 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:58,877 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:58,877 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:58,877 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:58,883 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,883 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,884 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:58,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:58,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:58,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:58,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:58,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:58,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:58,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:58,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,900 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:58,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:58,900 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:58,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:58,901 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:58,901 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:58,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:58,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:58,901 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,001 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,005 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,005 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,008 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,008 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,010 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,010 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,010 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,010 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,010 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,010 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,011 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,011 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,011 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,011 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,013 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,013 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,013 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@793482a8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,015 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,015 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,018 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,018 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,019 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,020 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,020 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,020 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,023 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,023 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,023 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,036 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,036 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,037 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,037 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695019065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695019069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695019069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695019078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,137 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,140 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,142 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,144 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,144 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,146 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c789738[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,150 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,151 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,153 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,153 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,156 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,156 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,158 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,161 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,166 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,166 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,166 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,167 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,267 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,270 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,276 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,277 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,277 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,280 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,281 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,281 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4c6649[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,283 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,285 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,286 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,286 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,289 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,289 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,290 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/866a36f4dafe4820b725b803398fd0a5 2023-03-24 21:55:59,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:59,299 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,299 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,300 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,399 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,403 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,405 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,406 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,407 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,407 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,407 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,407 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,407 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,409 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732e34f9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,414 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,415 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,417 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,418 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,422 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,422 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,425 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,435 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,435 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:59,535 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,539 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,539 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,540 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,542 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,543 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,544 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,544 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,544 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,544 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,544 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,545 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,550 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed2c9f6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,555 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,556 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,556 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,556 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,556 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,562 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,562 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,566 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,576 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,576 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,576 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48142 deadline: 1679695019573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48234 deadline: 1679695019574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48194 deadline: 1679695019578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:55:59,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:48196 deadline: 1679695019581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,676 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,680 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,680 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,682 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,682 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,683 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,683 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,683 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,683 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,685 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,685 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,685 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. as already flushing 2023-03-24 21:55:59,685 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:55:59,685 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:55:59,688 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:55:59,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@467b5a43[Count = 0] remaining members to acquire global barrier 2023-03-24 21:55:59,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:55:59,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:55:59,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:55:59,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:55:59,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:55:59,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:55:59,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:55:59,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:55:59,701 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:55:59,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:55:59,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:55:59,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:55:59,701 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:55:59,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:55:59,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:55:59,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:55:59,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:55:59,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:55:59,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:55:59,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:55:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,708 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:55:59,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:55:59,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:55:59,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:55:59,708 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:55:59,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:55:59,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,709 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=876 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/f57a00af6b514a50b10f027280796a08 2023-03-24 21:55:59,714 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e083da2215804cf1980a9db2ad9d78f7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e083da2215804cf1980a9db2ad9d78f7 2023-03-24 21:55:59,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e083da2215804cf1980a9db2ad9d78f7, entries=200, sequenceid=876, filesize=14.3 K 2023-03-24 21:55:59,717 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/866a36f4dafe4820b725b803398fd0a5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/866a36f4dafe4820b725b803398fd0a5 2023-03-24 21:55:59,721 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/866a36f4dafe4820b725b803398fd0a5, entries=150, sequenceid=876, filesize=11.9 K 2023-03-24 21:55:59,721 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/f57a00af6b514a50b10f027280796a08 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f57a00af6b514a50b10f027280796a08 2023-03-24 21:55:59,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f57a00af6b514a50b10f027280796a08, entries=150, sequenceid=876, filesize=11.9 K 2023-03-24 21:55:59,729 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 2a79b669e50783c6f283e5def38723af in 1312ms, sequenceid=876, compaction requested=false 2023-03-24 21:55:59,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:55:59,809 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:55:59,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:55:59,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:55:59,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:55:59,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:55:59,812 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:55:59,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:55:59,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:55:59,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:55:59,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:55:59,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:55:59,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:55:59,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:55:59,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:55:59,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:55:59,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:55:59,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:55:59,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:55:59,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:55:59,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:55:59,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:55:59,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:55:59,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:55:59,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. started... 2023-03-24 21:55:59,817 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:55:59,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:55:59,844 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e18930f6c38f4389b3ed48ec90a517ed 2023-03-24 21:56:00,270 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/58a1515ba73644a4ae4a704cfcda4045 2023-03-24 21:56:00,282 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/c24fc7f822c844a9b018cc33b1033d1e 2023-03-24 21:56:00,287 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/e18930f6c38f4389b3ed48ec90a517ed as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e18930f6c38f4389b3ed48ec90a517ed 2023-03-24 21:56:00,297 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e18930f6c38f4389b3ed48ec90a517ed, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 21:56:00,298 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/58a1515ba73644a4ae4a704cfcda4045 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/58a1515ba73644a4ae4a704cfcda4045 2023-03-24 21:56:00,306 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/58a1515ba73644a4ae4a704cfcda4045, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 21:56:00,308 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/c24fc7f822c844a9b018cc33b1033d1e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c24fc7f822c844a9b018cc33b1033d1e 2023-03-24 21:56:00,312 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c24fc7f822c844a9b018cc33b1033d1e, entries=150, sequenceid=892, filesize=11.9 K 2023-03-24 21:56:00,312 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 2a79b669e50783c6f283e5def38723af in 495ms, sequenceid=892, compaction requested=true 2023-03-24 21:56:00,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:56:00,313 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:56:00,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:00,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:00,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:00,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:00,315 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:00,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:00,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,315 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:00,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bf802eb[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:00,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:00,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:00,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:00,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:00,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:00,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:00,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:00,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:00,322 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:00,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:00,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:00,327 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:00,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:00,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:00,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:00,327 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:00,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:00,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:00,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:00,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,331 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:00,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,332 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,335 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:00,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:00,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:00,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:00,335 DEBUG [Thread-3044] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:00,335 DEBUG [Thread-3044] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:00,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:00,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:00,335 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:00,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:00,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:00,385 DEBUG [Thread-3051] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x19cafcd7 to 127.0.0.1:57724 2023-03-24 21:56:00,385 DEBUG [Thread-3051] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,386 DEBUG [Thread-3049] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c2afc3b to 127.0.0.1:57724 2023-03-24 21:56:00,386 DEBUG [Thread-3049] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,388 DEBUG [Thread-3053] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x37210381 to 127.0.0.1:57724 2023-03-24 21:56:00,388 DEBUG [Thread-3053] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,388 DEBUG [Thread-3045] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08344d0d to 127.0.0.1:57724 2023-03-24 21:56:00,388 DEBUG [Thread-3045] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,389 DEBUG [Thread-3047] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08ef5a5a to 127.0.0.1:57724 2023-03-24 21:56:00,389 DEBUG [Thread-3047] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,435 DEBUG [Thread-3044] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:00,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:00,583 DEBUG [Thread-3034] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x110b63af to 127.0.0.1:57724 2023-03-24 21:56:00,583 DEBUG [Thread-3034] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,584 DEBUG [Thread-3036] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bd64911 to 127.0.0.1:57724 2023-03-24 21:56:00,584 DEBUG [Thread-3036] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,589 DEBUG [Thread-3038] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6973f12f to 127.0.0.1:57724 2023-03-24 21:56:00,589 DEBUG [Thread-3038] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:00,591 DEBUG [Thread-3040] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1aac3cbf to 127.0.0.1:57724 2023-03-24 21:56:00,591 DEBUG [Thread-3040] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:06,917 DEBUG [Thread-3042] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1575f790 to 127.0.0.1:57724 2023-03-24 21:56:06,917 DEBUG [Thread-3042] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:06,917 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 229 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 207 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 126 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 90 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1634 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 4902 rows 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1649 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 4945 rows 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1640 2023-03-24 21:56:06,918 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 4918 rows 2023-03-24 21:56:06,919 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1643 2023-03-24 21:56:06,919 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 4928 rows 2023-03-24 21:56:06,919 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(402): scanned 1650 2023-03-24 21:56:06,919 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(403): verified 4950 rows 2023-03-24 21:56:06,919 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:56:06,919 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65df9e98 to 127.0.0.1:57724 2023-03-24 21:56:06,919 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:06,921 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:56:06,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:56:06,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:06,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:06,925 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694966925"}]},"ts":"1679694966925"} 2023-03-24 21:56:06,926 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:56:06,928 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:56:06,928 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, UNASSIGN}] 2023-03-24 21:56:06,930 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, UNASSIGN 2023-03-24 21:56:06,930 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2a79b669e50783c6f283e5def38723af, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:06,930 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694966930"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694966930"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694966930"}]},"ts":"1679694966930"} 2023-03-24 21:56:06,931 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:56:07,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:07,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 2a79b669e50783c6f283e5def38723af 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 2a79b669e50783c6f283e5def38723af, disabling compactions & flushes 2023-03-24 21:56:07,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. after waiting 0 ms 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:56:07,083 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 2a79b669e50783c6f283e5def38723af 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=A 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=B 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:07,083 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2a79b669e50783c6f283e5def38723af, store=C 2023-03-24 21:56:07,084 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:07,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:07,491 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d7fc89ea41a148b699d020bb8b951786 2023-03-24 21:56:07,499 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/775fdf24e3e84f3084011c75c8ec937e 2023-03-24 21:56:07,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:07,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=900 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/34f160c52ae54b0290c519f6c4eb1f74 2023-03-24 21:56:07,910 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/A/d7fc89ea41a148b699d020bb8b951786 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d7fc89ea41a148b699d020bb8b951786 2023-03-24 21:56:07,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d7fc89ea41a148b699d020bb8b951786, entries=150, sequenceid=900, filesize=11.9 K 2023-03-24 21:56:07,914 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/B/775fdf24e3e84f3084011c75c8ec937e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/775fdf24e3e84f3084011c75c8ec937e 2023-03-24 21:56:07,917 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/775fdf24e3e84f3084011c75c8ec937e, entries=150, sequenceid=900, filesize=11.9 K 2023-03-24 21:56:07,917 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/.tmp/C/34f160c52ae54b0290c519f6c4eb1f74 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34f160c52ae54b0290c519f6c4eb1f74 2023-03-24 21:56:07,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34f160c52ae54b0290c519f6c4eb1f74, entries=150, sequenceid=900, filesize=11.9 K 2023-03-24 21:56:07,921 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 2a79b669e50783c6f283e5def38723af in 838ms, sequenceid=900, compaction requested=true 2023-03-24 21:56:07,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1a70e6f2de7f48b783faa78711dd99e7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6b0e7e4c166b44f2967aa1729ac96293, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b0c5fd81584b425793ccad087da24f96, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1fd29c8cd97d4679904309abca4d6398, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/01c2e2a98e314abcb9904f602c9131a2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d048869c194e4137ad2315e5500299f0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b9f8165d7d1d47e8a2d862e3ef5233fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/7d73818d72a74b69bcdc5ef0689a715d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5a8d671e078a4be7b1031a7b899b7e4b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/63354df29448463d88fee01ef507477b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440] to archive 2023-03-24 21:56:07,947 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:07,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/95dea600d70e437dbdcd2f57d7508e51 2023-03-24 21:56:07,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/37d94f94ed9b47739682e1f7085fe16a 2023-03-24 21:56:07,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/195c489c149f485792a3e9c1223f8bfe 2023-03-24 21:56:07,951 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1a70e6f2de7f48b783faa78711dd99e7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1a70e6f2de7f48b783faa78711dd99e7 2023-03-24 21:56:07,952 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b91a8fbf9a13434b9711aedab7e64e0a 2023-03-24 21:56:07,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/87e53ed674dc4fddbf58daf19ae053ab 2023-03-24 21:56:07,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6b0e7e4c166b44f2967aa1729ac96293 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6b0e7e4c166b44f2967aa1729ac96293 2023-03-24 21:56:07,954 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/abf86d959d024ee28e05cffd8ee49e2d 2023-03-24 21:56:07,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cdd087ef5dda4c85bbcda9c6979f62d1 2023-03-24 21:56:07,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/629c76a26a6d40a298cbb848e59c3e1f 2023-03-24 21:56:07,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b0c5fd81584b425793ccad087da24f96 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b0c5fd81584b425793ccad087da24f96 2023-03-24 21:56:07,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/cba8f88684d141248cc68e2447c34523 2023-03-24 21:56:07,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/77652401d79e46628d3f1e20e7a57e0e 2023-03-24 21:56:07,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1b1407e08a6249a2896182f101966e5e 2023-03-24 21:56:07,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1fd29c8cd97d4679904309abca4d6398 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/1fd29c8cd97d4679904309abca4d6398 2023-03-24 21:56:07,960 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/11972ed817e04606a68d7e90cf6c108c 2023-03-24 21:56:07,961 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d92ff826161e49a996f94ab28f56536a 2023-03-24 21:56:07,962 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d46291d816e6413f822890de8dc3e82d 2023-03-24 21:56:07,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/01c2e2a98e314abcb9904f602c9131a2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/01c2e2a98e314abcb9904f602c9131a2 2023-03-24 21:56:07,964 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/8f604e62efc842ada15f0ebc4fde70fd 2023-03-24 21:56:07,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/75c76ea87f834985be36540643676819 2023-03-24 21:56:07,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d048869c194e4137ad2315e5500299f0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d048869c194e4137ad2315e5500299f0 2023-03-24 21:56:07,966 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5c80e2ad52d14558bd050accfcd1f764 2023-03-24 21:56:07,967 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5bf2912129d14039bc2700f08f0902a0 2023-03-24 21:56:07,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b9f8165d7d1d47e8a2d862e3ef5233fd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/b9f8165d7d1d47e8a2d862e3ef5233fd 2023-03-24 21:56:07,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/26ca11f024bc4aee9633604ace55ed3d 2023-03-24 21:56:07,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/947bf4aa47ee441e9e37981c6bda5a55 2023-03-24 21:56:07,970 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/89b0c305b3044c4a87c993988924c05e 2023-03-24 21:56:07,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/daaae95fa5a24f34adc90f95482607af 2023-03-24 21:56:07,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/7d73818d72a74b69bcdc5ef0689a715d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/7d73818d72a74b69bcdc5ef0689a715d 2023-03-24 21:56:07,972 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/025d5c096aa340b6abd3ff2acd5a8e09 2023-03-24 21:56:07,973 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/45348c929f6b445eb45d9b91db362080 2023-03-24 21:56:07,974 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5a8d671e078a4be7b1031a7b899b7e4b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/5a8d671e078a4be7b1031a7b899b7e4b 2023-03-24 21:56:07,974 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/35954058edda492dad94d515d96cb758 2023-03-24 21:56:07,975 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/63354df29448463d88fee01ef507477b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/63354df29448463d88fee01ef507477b 2023-03-24 21:56:07,976 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/bc4d7e865b5e4a5a9570fca5b4a158bc 2023-03-24 21:56:07,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/6cdd44853148478f8500d6ce0c24f267 2023-03-24 21:56:07,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e043a50b88c44ef68c14e5268dd4b601 2023-03-24 21:56:07,978 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/c73cfce0e235492f9aa34cdd357e0440 2023-03-24 21:56:07,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8469ed95fdf34189a78d383a96c0a225, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/1d1bd750d501493a8d32da93a3a22058, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5acff26721e94e4d8599cb596188bdbc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/f03d072722da475f85d3cab404a44dc1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2974f7b2babe47069127f5e7c9985a9b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/bc71785248aa4308ad5c69fd7a65b076, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/3592872d9a3940cbbe9ea9d509d42bf6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/72bd9d81e612489b8ae3c88340e9432a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d2fd8294ba7b46b5930b545b47cb5d8c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2440fae244714b159ff397eee5c1299d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673] to archive 2023-03-24 21:56:07,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:07,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/dfad0585b41b4f318423845439d89de7 2023-03-24 21:56:07,984 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/673bd2f68d0944aebdb3a523189728eb 2023-03-24 21:56:07,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8469ed95fdf34189a78d383a96c0a225 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8469ed95fdf34189a78d383a96c0a225 2023-03-24 21:56:07,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/010b16c47fad42b698ae9620478b3750 2023-03-24 21:56:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/8aa7944435f049978759b18f3a2640cb 2023-03-24 21:56:07,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/1d1bd750d501493a8d32da93a3a22058 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/1d1bd750d501493a8d32da93a3a22058 2023-03-24 21:56:07,988 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c823122e496a427da7ded49ec2b1b2e5 2023-03-24 21:56:07,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/9e5c44bec3b7411cb6fda2a183e296c3 2023-03-24 21:56:07,990 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/c97b3d197b944b21bd45a137be20c987 2023-03-24 21:56:07,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5acff26721e94e4d8599cb596188bdbc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5acff26721e94e4d8599cb596188bdbc 2023-03-24 21:56:07,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ac0efc6da5d4568b5c83e661cf929cc 2023-03-24 21:56:07,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/99a9eab9eb2a4de29347f0fbec5903aa 2023-03-24 21:56:07,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fe749462862b4361bb3ba2425f8ee546 2023-03-24 21:56:07,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/45f2c28eecae4791959cacf059c2c898 2023-03-24 21:56:07,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/f03d072722da475f85d3cab404a44dc1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/f03d072722da475f85d3cab404a44dc1 2023-03-24 21:56:07,995 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/985c44d9910046dab748bab4fb2e2345 2023-03-24 21:56:07,996 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4f0141922ad24d9aad6c3c6f8c3841a2 2023-03-24 21:56:07,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2974f7b2babe47069127f5e7c9985a9b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2974f7b2babe47069127f5e7c9985a9b 2023-03-24 21:56:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/938dfaa82f83495b90c01d2851eaa952 2023-03-24 21:56:07,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/e19d96fc381c4229837d6631b64cd486 2023-03-24 21:56:07,999 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/bc71785248aa4308ad5c69fd7a65b076 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/bc71785248aa4308ad5c69fd7a65b076 2023-03-24 21:56:08,000 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/6ca7e58bb3a344988135923483f05f97 2023-03-24 21:56:08,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/91647d6d52a44ddea304a12a6c2cc757 2023-03-24 21:56:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/3592872d9a3940cbbe9ea9d509d42bf6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/3592872d9a3940cbbe9ea9d509d42bf6 2023-03-24 21:56:08,002 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/15efc92cdf954bcdb8a8aa10e36f5ad7 2023-03-24 21:56:08,003 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/01fc86683c6645cbab8987dd43318f69 2023-03-24 21:56:08,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d341433e6c6848eb8d303916020a4b35 2023-03-24 21:56:08,005 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d1d382ae3eb2410bacf03a08be33aa01 2023-03-24 21:56:08,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/72bd9d81e612489b8ae3c88340e9432a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/72bd9d81e612489b8ae3c88340e9432a 2023-03-24 21:56:08,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/aae792c8beb1402383c1a680d6c7a107 2023-03-24 21:56:08,007 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2a3964896f214a76a6ade891b20d37a3 2023-03-24 21:56:08,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d2fd8294ba7b46b5930b545b47cb5d8c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/d2fd8294ba7b46b5930b545b47cb5d8c 2023-03-24 21:56:08,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/4c2ca94b54e54f81acd17a7e165031d4 2023-03-24 21:56:08,010 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5a04d8b2749a484c9b1c9928c8f35196 2023-03-24 21:56:08,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2440fae244714b159ff397eee5c1299d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2440fae244714b159ff397eee5c1299d 2023-03-24 21:56:08,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5424f4549b90455eb15f8273064a5928 2023-03-24 21:56:08,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/5f3e6c11dddc46128a80825f4ca019b4 2023-03-24 21:56:08,014 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/84ab6131692c43a2a075a66675507cf5 2023-03-24 21:56:08,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/2e201cb4516543ba96ae46c6ddeaf673 2023-03-24 21:56:08,018 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/11e7cc040d704711b96ace26e5d98833, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/0843f8f81415411faf73ae7076df73ec, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/95cf72811c3742578db21c2270738832, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6c495d15fc47446fb1ab2207678266c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b37851735c67425f9f66460c3f269a24, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2bc7d11d6056451d9e106437196d2a3e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e52927a31c394ec3893fc0134256905e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/fd208bcd70ea49b7a49d0f751e15a173, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7dc7b291e1e8403eaf33feece10122c8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7229e71d55b94262b07c19a6b3dff296, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb] to archive 2023-03-24 21:56:08,019 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:08,020 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e0fa8904fa4e4c4aa65117d2dad5cbcd 2023-03-24 21:56:08,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad68c47ac8d74432b3d65dcbaceea407 2023-03-24 21:56:08,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/11e7cc040d704711b96ace26e5d98833 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/11e7cc040d704711b96ace26e5d98833 2023-03-24 21:56:08,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34663e532a0048528e6c15f7d43ce27d 2023-03-24 21:56:08,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2792ec870f89415388e0755df3711533 2023-03-24 21:56:08,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/0843f8f81415411faf73ae7076df73ec to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/0843f8f81415411faf73ae7076df73ec 2023-03-24 21:56:08,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/dde3df054da74eb9984a982d7e2ca640 2023-03-24 21:56:08,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:08,028 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/121e08498fba4958bf064a56c8f20093 2023-03-24 21:56:08,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/a3d0c04e0ae84e8080f952d64eab301c 2023-03-24 21:56:08,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/95cf72811c3742578db21c2270738832 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/95cf72811c3742578db21c2270738832 2023-03-24 21:56:08,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/973245f177ff4941818230109b3addc4 2023-03-24 21:56:08,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/d7aaf9140a924f50ab2fb22a39dedf58 2023-03-24 21:56:08,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7ed0ebd39ef54692a81a16f10a89ae02 2023-03-24 21:56:08,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b94bde939e6c4e3d8b13b441c6ca4d2a 2023-03-24 21:56:08,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6c495d15fc47446fb1ab2207678266c9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6c495d15fc47446fb1ab2207678266c9 2023-03-24 21:56:08,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ee7395e4a2de4dc58a6f861a466b941f 2023-03-24 21:56:08,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/33694e288c5544108ab4e09e8afed037 2023-03-24 21:56:08,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b37851735c67425f9f66460c3f269a24 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/b37851735c67425f9f66460c3f269a24 2023-03-24 21:56:08,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/91451afa96fb4933b6faf3ccb1a1de50 2023-03-24 21:56:08,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/260c8de726d0424f9d185d815932164d 2023-03-24 21:56:08,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2bc7d11d6056451d9e106437196d2a3e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2bc7d11d6056451d9e106437196d2a3e 2023-03-24 21:56:08,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e21ebea8c18e4fb581386918ce108b78 2023-03-24 21:56:08,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/850c85704f174d108b916db567bd98ef 2023-03-24 21:56:08,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e52927a31c394ec3893fc0134256905e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e52927a31c394ec3893fc0134256905e 2023-03-24 21:56:08,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f6239f2ad40547819d48e498390415f8 2023-03-24 21:56:08,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/2004eaa48eed444da81c3db74f655f53 2023-03-24 21:56:08,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/3eb8988dd80c42e1831f1aaae3c25087 2023-03-24 21:56:08,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/ad83e88568a6465c92c55e263e41df99 2023-03-24 21:56:08,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/fd208bcd70ea49b7a49d0f751e15a173 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/fd208bcd70ea49b7a49d0f751e15a173 2023-03-24 21:56:08,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/6e4c287771b44f61b9541b403279e6dc 2023-03-24 21:56:08,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/43abee2cfc9544fd95cdb666d5c278f8 2023-03-24 21:56:08,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7dc7b291e1e8403eaf33feece10122c8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7dc7b291e1e8403eaf33feece10122c8 2023-03-24 21:56:08,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c7d629f2c7b84239a72db248b840d8ac 2023-03-24 21:56:08,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/4c0a6d3699eb4afc9c26bb263ae422c8 2023-03-24 21:56:08,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/e2fdfe09a8024ce490a9c1d376748ac9 2023-03-24 21:56:08,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7229e71d55b94262b07c19a6b3dff296 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/7229e71d55b94262b07c19a6b3dff296 2023-03-24 21:56:08,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/da66716104fa4954a9bc32cce5ab1609 2023-03-24 21:56:08,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/00292e388841411fb538caf0529572f5 2023-03-24 21:56:08,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/23bd1d2229a1480fa6717d65f295bcfb 2023-03-24 21:56:08,062 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/recovered.edits/903.seqid, newMaxSeqId=903, maxSeqId=1 2023-03-24 21:56:08,063 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af. 2023-03-24 21:56:08,063 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 2a79b669e50783c6f283e5def38723af: 2023-03-24 21:56:08,064 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 2a79b669e50783c6f283e5def38723af 2023-03-24 21:56:08,065 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=2a79b669e50783c6f283e5def38723af, regionState=CLOSED 2023-03-24 21:56:08,065 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694968065"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694968065"}]},"ts":"1679694968065"} 2023-03-24 21:56:08,066 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-24 21:56:08,067 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 2a79b669e50783c6f283e5def38723af, server=jenkins-hbase9.apache.org,45673,1679694819052 in 1.1350 sec 2023-03-24 21:56:08,067 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-24 21:56:08,068 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2a79b669e50783c6f283e5def38723af, UNASSIGN in 1.1390 sec 2023-03-24 21:56:08,068 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694968068"}]},"ts":"1679694968068"} 2023-03-24 21:56:08,069 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:56:08,071 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:56:08,072 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1510 sec 2023-03-24 21:56:09,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-24 21:56:09,030 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-24 21:56:09,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:56:09,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,032 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 21:56:09,033 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,036 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:56:09,038 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/recovered.edits] 2023-03-24 21:56:09,041 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/0ae30b6c928d44e282a6bdb7c2ec7a33 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/0ae30b6c928d44e282a6bdb7c2ec7a33 2023-03-24 21:56:09,042 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d7fc89ea41a148b699d020bb8b951786 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/d7fc89ea41a148b699d020bb8b951786 2023-03-24 21:56:09,043 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e083da2215804cf1980a9db2ad9d78f7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e083da2215804cf1980a9db2ad9d78f7 2023-03-24 21:56:09,044 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e18930f6c38f4389b3ed48ec90a517ed to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/A/e18930f6c38f4389b3ed48ec90a517ed 2023-03-24 21:56:09,046 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/58a1515ba73644a4ae4a704cfcda4045 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/58a1515ba73644a4ae4a704cfcda4045 2023-03-24 21:56:09,047 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/775fdf24e3e84f3084011c75c8ec937e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/775fdf24e3e84f3084011c75c8ec937e 2023-03-24 21:56:09,048 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/866a36f4dafe4820b725b803398fd0a5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/866a36f4dafe4820b725b803398fd0a5 2023-03-24 21:56:09,049 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fa9c3488bf90422dbe3762bad90941da to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/B/fa9c3488bf90422dbe3762bad90941da 2023-03-24 21:56:09,051 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34f160c52ae54b0290c519f6c4eb1f74 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/34f160c52ae54b0290c519f6c4eb1f74 2023-03-24 21:56:09,051 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c24fc7f822c844a9b018cc33b1033d1e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/c24fc7f822c844a9b018cc33b1033d1e 2023-03-24 21:56:09,052 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f03e48a5daf640f98aea705421be4add to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f03e48a5daf640f98aea705421be4add 2023-03-24 21:56:09,053 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f57a00af6b514a50b10f027280796a08 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/C/f57a00af6b514a50b10f027280796a08 2023-03-24 21:56:09,055 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/recovered.edits/903.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af/recovered.edits/903.seqid 2023-03-24 21:56:09,056 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/2a79b669e50783c6f283e5def38723af 2023-03-24 21:56:09,056 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:56:09,058 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,059 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:56:09,060 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:56:09,061 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,061 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:56:09,061 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694969061"}]},"ts":"9223372036854775807"} 2023-03-24 21:56:09,063 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:56:09,063 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2a79b669e50783c6f283e5def38723af, NAME => 'TestAcidGuarantees,,1679694938218.2a79b669e50783c6f283e5def38723af.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:56:09,063 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:56:09,063 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694969063"}]},"ts":"9223372036854775807"} 2023-03-24 21:56:09,064 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:56:09,066 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-03-24 21:56:09,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-24 21:56:09,134 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-24 21:56:09,147 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 227), OpenFileDescriptor=586 (was 544) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=446 (was 363) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8286 (was 7853) - AvailableMemoryMB LEAK? - 2023-03-24 21:56:09,160 INFO [Listener at localhost/40745] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=586, MaxFileDescriptor=60000, SystemLoadAverage=446, ProcessCount=169, AvailableMemoryMB=8285 2023-03-24 21:56:09,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:56:09,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$4(2112): Client=jenkins//172.31.2.10 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:56:09,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:09,164 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-24 21:56:09,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(697): Client=jenkins//172.31.2.10 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-24 21:56:09,165 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-24 21:56:09,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:09,166 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:09,166 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a empty. 2023-03-24 21:56:09,167 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:09,167 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:56:09,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:09,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:09,573 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:56:09,575 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 09571a8f4d95ff573f25f32562564d3a, NAME => 'TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp 2023-03-24 21:56:09,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 09571a8f4d95ff573f25f32562564d3a, disabling compactions & flushes 2023-03-24 21:56:09,980 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. after waiting 0 ms 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:09,980 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:09,980 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:09,982 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-24 21:56:09,983 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694969983"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694969983"}]},"ts":"1679694969983"} 2023-03-24 21:56:09,984 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-24 21:56:09,984 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-24 21:56:09,984 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694969984"}]},"ts":"1679694969984"} 2023-03-24 21:56:09,985 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-24 21:56:09,989 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, ASSIGN}] 2023-03-24 21:56:09,990 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, ASSIGN 2023-03-24 21:56:09,991 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, ASSIGN; state=OFFLINE, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=false 2023-03-24 21:56:10,142 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:10,142 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694970142"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694970142"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694970142"}]},"ts":"1679694970142"} 2023-03-24 21:56:10,144 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:56:10,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:10,299 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 09571a8f4d95ff573f25f32562564d3a, NAME => 'TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:56:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:56:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,299 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,301 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,302 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A 2023-03-24 21:56:10,302 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A 2023-03-24 21:56:10,302 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:10,302 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName A 2023-03-24 21:56:10,303 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:10,303 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,304 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B 2023-03-24 21:56:10,304 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B 2023-03-24 21:56:10,304 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:10,304 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName B 2023-03-24 21:56:10,305 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:10,305 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,306 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C 2023-03-24 21:56:10,306 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C 2023-03-24 21:56:10,306 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:10,306 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName C 2023-03-24 21:56:10,306 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:10,306 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:10,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,307 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,309 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:56:10,310 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:10,311 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-24 21:56:10,312 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 09571a8f4d95ff573f25f32562564d3a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61220777, jitterRate=-0.08773933351039886}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:56:10,312 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:10,313 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., pid=56, masterSystemTime=1679694970295 2023-03-24 21:56:10,314 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:10,314 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:10,314 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:10,314 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694970314"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694970314"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694970314"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694970314"}]},"ts":"1679694970314"} 2023-03-24 21:56:10,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-24 21:56:10,316 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 in 172 msec 2023-03-24 21:56:10,317 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-24 21:56:10,317 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, ASSIGN in 327 msec 2023-03-24 21:56:10,318 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-24 21:56:10,318 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694970318"}]},"ts":"1679694970318"} 2023-03-24 21:56:10,318 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-24 21:56:10,320 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-24 21:56:10,321 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-03-24 21:56:11,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-24 21:56:11,269 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-24 21:56:11,270 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x649a2bc4 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:11,275 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fe6f970, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:11,276 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:11,277 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56530, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:11,278 DEBUG [Listener at localhost/40745] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-24 21:56:11,279 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:44952, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-24 21:56:11,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-24 21:56:11,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$12(2454): Client=jenkins//172.31.2.10 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-24 21:56:11,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:11,694 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:56:11,695 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-24 21:56:11,695 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-24 21:56:11,697 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-24 21:56:11,699 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, REOPEN/MOVE}] 2023-03-24 21:56:11,699 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, REOPEN/MOVE 2023-03-24 21:56:11,700 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:11,700 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694971700"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694971700"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694971700"}]},"ts":"1679694971700"} 2023-03-24 21:56:11,701 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:56:11,852 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:11,853 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 09571a8f4d95ff573f25f32562564d3a, disabling compactions & flushes 2023-03-24 21:56:11,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:11,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:11,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. after waiting 0 ms 2023-03-24 21:56:11,854 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:11,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-24 21:56:11,859 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:11,859 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:11,859 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 09571a8f4d95ff573f25f32562564d3a to self. 2023-03-24 21:56:11,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:11,860 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=CLOSED 2023-03-24 21:56:11,860 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694971860"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694971860"}]},"ts":"1679694971860"} 2023-03-24 21:56:11,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-24 21:56:11,862 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 in 160 msec 2023-03-24 21:56:11,862 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase9.apache.org,45673,1679694819052; forceNewPlan=false, retain=true 2023-03-24 21:56:12,013 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=OPENING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,013 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694972013"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694972013"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694972013"}]},"ts":"1679694972013"} 2023-03-24 21:56:12,014 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:56:12,169 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,169 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 09571a8f4d95ff573f25f32562564d3a, NAME => 'TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.', STARTKEY => '', ENDKEY => ''} 2023-03-24 21:56:12,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-24 21:56:12,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7894): checking encryption for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,170 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(7897): checking classloading for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,171 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,172 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A 2023-03-24 21:56:12,172 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A 2023-03-24 21:56:12,172 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:12,172 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName A 2023-03-24 21:56:12,174 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:12,174 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,175 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B 2023-03-24 21:56:12,175 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B 2023-03-24 21:56:12,175 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:12,175 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName B 2023-03-24 21:56:12,176 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:12,176 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,176 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C 2023-03-24 21:56:12,176 DEBUG [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C 2023-03-24 21:56:12,176 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-24 21:56:12,177 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 09571a8f4d95ff573f25f32562564d3a columnFamilyName C 2023-03-24 21:56:12,177 INFO [StoreOpener-09571a8f4d95ff573f25f32562564d3a-1] regionserver.HStore(310): Store=09571a8f4d95ff573f25f32562564d3a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-24 21:56:12,177 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,178 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,180 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-24 21:56:12,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1055): writing seq id for 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,181 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1072): Opened 09571a8f4d95ff573f25f32562564d3a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=70691797, jitterRate=0.053389862179756165}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-24 21:56:12,181 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(965): Region open journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:12,182 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., pid=61, masterSystemTime=1679694972165 2023-03-24 21:56:12,185 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,185 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694972185"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679694972185"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679694972185"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679694972185"}]},"ts":"1679694972185"} 2023-03-24 21:56:12,186 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,186 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase9:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-24 21:56:12,187 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 in 172 msec 2023-03-24 21:56:12,188 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-24 21:56:12,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, REOPEN/MOVE in 488 msec 2023-03-24 21:56:12,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-24 21:56:12,190 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 492 msec 2023-03-24 21:56:12,192 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 910 msec 2023-03-24 21:56:12,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-24 21:56:12,193 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x5f059091 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,198 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a7444be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,199 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x62f836a7 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,203 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@118ab371, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,203 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x43c7d444 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,207 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60a99f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,207 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x670e081b to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,210 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1a16954a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,211 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x779c2944 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,214 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@60c5a97c, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,214 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x11947498 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,222 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cd7b19f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,223 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x0e38d584 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,228 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cba406f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,228 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x4df4e4ba to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,237 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5289c75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,237 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x34090ce3 to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,240 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5eb2efec, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,241 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(139): Connect 0x0c382e3e to 127.0.0.1:57724 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-24 21:56:12,245 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@702a554e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-24 21:56:12,249 DEBUG [hconnection-0x1646779e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,250 DEBUG [hconnection-0x2c480386-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,251 DEBUG [hconnection-0x37a82022-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,252 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56564, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,255 DEBUG [hconnection-0x90f9bd1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,259 DEBUG [hconnection-0x4252e301-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,260 DEBUG [hconnection-0x1c443cd9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,260 DEBUG [hconnection-0x64634d16-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,263 DEBUG [hconnection-0x53f7fd54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,264 DEBUG [hconnection-0x59668b65-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,264 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56554, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,264 DEBUG [hconnection-0x586ea5cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-24 21:56:12,264 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56540, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,265 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56582, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,265 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56572, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,266 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56598, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,271 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56574, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,273 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56586, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:12,275 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56606, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,275 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.2.10:56590, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-24 21:56:12,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:12,284 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:12,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:12,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:12,287 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:12,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:12,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:12,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:12,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:12,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:12,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:12,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:12,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:12,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694972298, max timestamp=1679694972302Num uniques 859; 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694972298, max timestamp=1679694972302Num uniques 859; 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694972298, max timestamp=1679694972302Num uniques 859; 2023-03-24 21:56:12,303 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:12,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:12,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:12,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:12,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:12,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:12,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39f36091[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:12,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:12,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:12,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:12,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,311 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,312 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:12,312 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,312 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324da1aeed0cdb14e6ebc702477938e1f65 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324da1aeed0cdb14e6ebc702477938e1f65 2023-03-24 21:56:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:12,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:12,316 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:12,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:12,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:12,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:12,316 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:12,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,318 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,319 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,319 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/5d6ceb02e7354be092abcec3d5e84e4c 2023-03-24 21:56:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,323 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:12,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:12,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,323 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,324 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:12,324 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:12,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695032324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695032325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695032326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695032326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695032327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,424 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:12,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:12,437 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:12,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:12,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:12,438 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:12,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:12,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:12,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,439 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695032436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695032436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695032436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695032437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695032437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,443 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:12,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:12,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:12,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:12,445 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:12,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:12,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:12,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d68320[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:12,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:12,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,449 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,449 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:12,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:12,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:12,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:12,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:12,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:12,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:12,453 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:12,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:12,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:12,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:12,453 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:12,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,457 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,458 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,460 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:12,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:12,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,460 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:12,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,461 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:12,561 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:12,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:12,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:12,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:12,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:12,564 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:12,564 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:12,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:12,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,565 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,567 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,568 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:12,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:12,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:12,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:12,569 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:12,570 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,570 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:12,570 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:12,570 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:12,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:12,572 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,572 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:12,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de3781b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:12,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:12,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:12,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:12,577 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:12,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:12,578 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:12,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,578 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:12,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:12,581 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:12,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:12,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:12,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:12,581 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:12,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:12,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:12,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,588 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:12,588 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:12,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,642 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695032640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695032641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695032641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695032641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:12,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695032642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,688 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:12,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:12,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:12,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:12,691 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:12,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:12,691 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:12,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,692 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:12,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:12,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:12,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:12,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:12,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:12,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:12,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:12,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:12,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:12,703 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:12,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:12,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:12,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:12,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:12,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6677ce16[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:12,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:12,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:12,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:12,714 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,714 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:12,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:12,716 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:12,716 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,717 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:12,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:12,719 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:12,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:12,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:12,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:12,719 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:12,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,721 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,722 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:12,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:12,725 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:12,725 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:12,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5cad76ab539849589a4ec2cf41a16fc6 2023-03-24 21:56:12,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/775dfc1100ef4142afcb382975ccd248 2023-03-24 21:56:12,760 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/5d6ceb02e7354be092abcec3d5e84e4c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c 2023-03-24 21:56:12,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c, entries=150, sequenceid=15, filesize=25.1 K 2023-03-24 21:56:12,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5cad76ab539849589a4ec2cf41a16fc6 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6 2023-03-24 21:56:12,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 21:56:12,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/775dfc1100ef4142afcb382975ccd248 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248 2023-03-24 21:56:12,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248, entries=150, sequenceid=15, filesize=11.6 K 2023-03-24 21:56:12,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 09571a8f4d95ff573f25f32562564d3a in 489ms, sequenceid=15, compaction requested=false 2023-03-24 21:56:12,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:12,825 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:12,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:12,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:12,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:12,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:12,831 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:12,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:12,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:12,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,833 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:12,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:12,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:12,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:12,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:12,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:12,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:12,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:12,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:12,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,834 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:12,835 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:12,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:12,849 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324384d257846ed4023b2e6b3f63c2a9400 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324384d257846ed4023b2e6b3f63c2a9400 2023-03-24 21:56:12,861 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/4577419fd80a4b15b272e0ba0a5c438c 2023-03-24 21:56:12,871 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/6186129dc01243e7ad4b2b48f1a73ad9 2023-03-24 21:56:12,889 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/b70abf8fe28e475287ebc9978c72ffe3 2023-03-24 21:56:12,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/4577419fd80a4b15b272e0ba0a5c438c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c 2023-03-24 21:56:12,895 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c, entries=150, sequenceid=52, filesize=25.1 K 2023-03-24 21:56:12,895 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/6186129dc01243e7ad4b2b48f1a73ad9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9 2023-03-24 21:56:12,898 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9, entries=150, sequenceid=52, filesize=11.6 K 2023-03-24 21:56:12,899 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/b70abf8fe28e475287ebc9978c72ffe3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3 2023-03-24 21:56:12,905 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3, entries=150, sequenceid=52, filesize=11.6 K 2023-03-24 21:56:12,905 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 70ms, sequenceid=52, compaction requested=false 2023-03-24 21:56:12,906 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:12,906 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:12,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:12,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:12,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:12,906 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:12,908 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,908 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,909 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:12,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4422dda1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:12,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:12,910 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,911 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:12,911 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,912 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:12,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:12,912 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:12,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:12,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:12,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:12,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:12,916 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:12,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:12,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:12,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:12,916 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:12,917 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,917 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,917 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:12,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,918 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:12,919 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:12,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:12,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:12,923 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:12,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:12,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:12,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:12,922 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:12,923 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:12,923 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:12,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:12,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:12,923 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:12,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:12,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:12,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:13,007 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,008 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694972979, max timestamp=1679694973006Num uniques 859; 2023-03-24 21:56:13,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694972979, max timestamp=1679694973006Num uniques 859; 2023-03-24 21:56:13,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:56:13,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679694972979, max timestamp=1679694973006Num uniques 859; 2023-03-24 21:56:13,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:56:13,009 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-24 21:56:13,023 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695033020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695033021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:13,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:13,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:13,027 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:13,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:13,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:13,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,028 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,029 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,029 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,029 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,029 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:13,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:13,030 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:13,030 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:13,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:13,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:13,031 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,031 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:13,031 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:13,031 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:13,031 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:13,032 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,032 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,033 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:13,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d88986[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:13,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:13,034 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,035 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:13,035 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:13,036 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:13,036 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:13,038 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,038 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:13,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:13,041 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:13,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:13,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:13,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:13,041 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:13,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,043 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,044 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,046 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,048 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:13,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,049 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:13,049 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:13,048 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,049 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695033125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695033126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,149 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:13,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:13,151 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:13,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:13,152 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:13,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:13,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:13,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:13,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,156 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:13,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:13,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:13,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:13,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:13,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:13,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:13,157 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:13,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:13,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,160 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:13,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77fd1406[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:13,162 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:13,162 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,163 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:13,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:13,163 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:13,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:13,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:13,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,167 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:13,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:13,170 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:13,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:13,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:13,170 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:13,170 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:13,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,172 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,173 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,174 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,178 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:13,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,178 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:13,178 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:13,179 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:13,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,279 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:13,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:13,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:13,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:13,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:13,282 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:13,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:13,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:13,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,289 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:13,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:13,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:13,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:13,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:13,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:13,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:13,290 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:13,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:13,290 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:13,291 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:13,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:13,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:13,291 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:13,292 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,292 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,293 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:13,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f227391[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:13,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:13,294 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,295 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:13,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:13,295 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:13,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,295 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:13,297 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:13,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,297 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:13,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:13,299 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:13,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:13,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:13,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:13,299 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:13,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,301 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,304 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:13,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:13,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,307 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,307 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:13,307 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,307 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695033329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,332 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695033330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,384 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324817160ef4b414199a00bdd85eff56859 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324817160ef4b414199a00bdd85eff56859 2023-03-24 21:56:13,390 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/706c85894f824bd3a76f79a7761a8561 2023-03-24 21:56:13,408 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:13,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:13,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:13,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:13,421 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:13,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:13,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:13,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,421 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:13,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:13,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,424 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:13,425 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:13,425 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:13,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:13,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:13,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:13,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:13,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/95384b13e0ad44c2af4327fd2e95c810 2023-03-24 21:56:13,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:13,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:13,427 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:13,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:13,427 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:13,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:13,433 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,433 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:13,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3f6c01[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:13,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:13,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,440 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,440 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:13,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:13,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:13,441 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:13,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:13,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:13,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:13,445 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:13,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:13,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:13,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:13,446 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:13,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:13,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:13,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,449 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/281113eb979d4b0296a01b08d3fc1096 2023-03-24 21:56:13,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:13,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:13,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:13,453 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:13,454 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,454 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:13,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:13,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:13,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:13,454 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:13,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/706c85894f824bd3a76f79a7761a8561 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561 2023-03-24 21:56:13,462 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561, entries=400, sequenceid=63, filesize=59.0 K 2023-03-24 21:56:13,463 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/95384b13e0ad44c2af4327fd2e95c810 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810 2023-03-24 21:56:13,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810, entries=150, sequenceid=63, filesize=11.6 K 2023-03-24 21:56:13,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/281113eb979d4b0296a01b08d3fc1096 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096 2023-03-24 21:56:13,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096, entries=150, sequenceid=63, filesize=11.6 K 2023-03-24 21:56:13,471 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09571a8f4d95ff573f25f32562564d3a in 505ms, sequenceid=63, compaction requested=true 2023-03-24 21:56:13,471 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:13,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:13,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:13,472 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:13,472 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:13,472 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 111946 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:13,473 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,473 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,473 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=34.8 K 2023-03-24 21:56:13,473 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=109.3 K 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5cad76ab539849589a4ec2cf41a16fc6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679694972277 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 5d6ceb02e7354be092abcec3d5e84e4c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679694972277 2023-03-24 21:56:13,473 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6186129dc01243e7ad4b2b48f1a73ad9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679694972319 2023-03-24 21:56:13,474 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4577419fd80a4b15b272e0ba0a5c438c, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679694972319 2023-03-24 21:56:13,474 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 95384b13e0ad44c2af4327fd2e95c810, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972960 2023-03-24 21:56:13,474 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 706c85894f824bd3a76f79a7761a8561, keycount=400, bloomtype=ROW, size=59.0 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972957 2023-03-24 21:56:13,479 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:13,490 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#516 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:13,494 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/a98898c77722416cbf39d509cb9b9486 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a98898c77722416cbf39d509cb9b9486 2023-03-24 21:56:13,497 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into a98898c77722416cbf39d509cb9b9486(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:13,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:13,498 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694973472; duration=0sec 2023-03-24 21:56:13,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:13,498 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:13,499 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:13,499 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:13,499 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,499 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=34.8 K 2023-03-24 21:56:13,499 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 775dfc1100ef4142afcb382975ccd248, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1679694972277 2023-03-24 21:56:13,500 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b70abf8fe28e475287ebc9978c72ffe3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679694972319 2023-03-24 21:56:13,500 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 281113eb979d4b0296a01b08d3fc1096, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972960 2023-03-24 21:56:13,505 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#517 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:13,554 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:13,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:13,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:13,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:13,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:13,557 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:13,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:13,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:13,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,560 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:13,560 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:13,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:13,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:13,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:13,561 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:13,561 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:13,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:13,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:13,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:13,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:13,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:13,562 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:13,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:13,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:13,562 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-24 21:56:13,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:13,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:13,562 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:13,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:13,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:13,563 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:13,585 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-24 21:56:13,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:13,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:13,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695033634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695033635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,742 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,917 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f33f95cda54e43c4896c8a635414f18f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f33f95cda54e43c4896c8a635414f18f 2023-03-24 21:56:13,921 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into f33f95cda54e43c4896c8a635414f18f(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:13,921 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:13,921 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694973472; duration=0sec 2023-03-24 21:56:13,922 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:13,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695033943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,945 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695033944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:13,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695033944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:13,976 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324335266e732d14b42b53ed4f9c026c46d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324335266e732d14b42b53ed4f9c026c46d 2023-03-24 21:56:13,982 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/0595d22b4e47443591fd54df4eec01be 2023-03-24 21:56:14,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695034140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,142 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695034140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695034247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,250 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695034247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695034247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,393 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/448e38d200954632aaf17c2c645b7e8f 2023-03-24 21:56:14,403 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/d5daebdc8d8a47ebae7446771c17a960 2023-03-24 21:56:14,406 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/0595d22b4e47443591fd54df4eec01be as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be 2023-03-24 21:56:14,409 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be, entries=150, sequenceid=98, filesize=25.1 K 2023-03-24 21:56:14,410 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/448e38d200954632aaf17c2c645b7e8f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f 2023-03-24 21:56:14,414 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 21:56:14,414 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/d5daebdc8d8a47ebae7446771c17a960 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960 2023-03-24 21:56:14,417 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960, entries=150, sequenceid=98, filesize=11.6 K 2023-03-24 21:56:14,418 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 09571a8f4d95ff573f25f32562564d3a in 856ms, sequenceid=98, compaction requested=false 2023-03-24 21:56:14,418 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:14,418 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:14,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:14,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:14,418 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:14,422 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:14,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,424 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:14,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,425 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:14,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:14,426 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@489e5e99[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:14,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:14,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,428 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:14,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:14,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:14,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,429 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:14,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:14,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:14,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:14,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:14,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:14,433 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:14,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:14,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:14,433 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:14,433 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:14,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:14,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:14,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:14,437 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:14,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:14,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:14,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:14,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:14,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:14,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:14,445 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:14,445 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:14,545 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:14,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:14,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:14,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:14,549 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:14,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:14,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:14,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:14,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,550 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:14,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:14,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:14,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:14,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:14,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:14,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:14,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:14,552 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:14,552 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:14,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:14,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:14,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:14,552 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:14,553 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:14,553 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:14,553 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:14,576 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242a8bc74cc1894ceeacce1aede92071d4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a8bc74cc1894ceeacce1aede92071d4 2023-03-24 21:56:14,583 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/c4e0216dd0cf49f39fb902a65956fec4 2023-03-24 21:56:14,714 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/7878085a61ad4b01bdcfc9debb890689 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7878085a61ad4b01bdcfc9debb890689 2023-03-24 21:56:14,718 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 7878085a61ad4b01bdcfc9debb890689(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-24 21:56:14,718 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:14,718 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694973471; duration=1sec 2023-03-24 21:56:14,718 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:14,753 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:14,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694974769, max timestamp=1679694974774Num uniques 859; 2023-03-24 21:56:14,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695034772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:14,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694974769, max timestamp=1679694974774Num uniques 859; 2023-03-24 21:56:14,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694974769, max timestamp=1679694974774Num uniques 859; 2023-03-24 21:56:14,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:14,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:14,799 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695034796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695034799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695034877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695034900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,903 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:14,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695034902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:14,992 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/0893d7e943214e548ccd6e27841ee654 2023-03-24 21:56:15,005 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/03ea24a9664149df832e03406a3ea3bc 2023-03-24 21:56:15,008 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/c4e0216dd0cf49f39fb902a65956fec4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4 2023-03-24 21:56:15,012 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4, entries=150, sequenceid=110, filesize=25.1 K 2023-03-24 21:56:15,013 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/0893d7e943214e548ccd6e27841ee654 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654 2023-03-24 21:56:15,016 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654, entries=150, sequenceid=110, filesize=11.6 K 2023-03-24 21:56:15,017 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/03ea24a9664149df832e03406a3ea3bc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc 2023-03-24 21:56:15,020 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc, entries=150, sequenceid=110, filesize=11.6 K 2023-03-24 21:56:15,021 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09571a8f4d95ff573f25f32562564d3a in 468ms, sequenceid=110, compaction requested=true 2023-03-24 21:56:15,021 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:15,021 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,023 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57dc3ea8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,024 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,026 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,027 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,031 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,031 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,031 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,037 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,037 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,037 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,038 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,041 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,041 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,041 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:15,082 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:15,083 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:15,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695035105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,108 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695035105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695035109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,142 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,145 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,147 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695035145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,147 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,148 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,149 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,149 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,149 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,150 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,150 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b5e6139[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,152 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695035151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,156 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,156 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,157 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,160 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,160 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,161 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,162 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,163 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,165 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,165 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,166 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,166 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,211 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695035209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,266 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,270 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,275 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,276 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,278 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,278 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d074737[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,281 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,282 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,282 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,284 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,284 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,287 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,287 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,287 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,290 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,290 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,291 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,292 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,293 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,298 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,299 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,299 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,399 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,402 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,406 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,406 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,406 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,406 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,408 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,408 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,409 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5757a698[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,411 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,411 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695035409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,412 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,412 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,412 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,413 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,413 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,413 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,413 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695035412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,416 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695035414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,416 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,416 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,418 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,418 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,418 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,420 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,423 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,423 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,424 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,424 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,424 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,497 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240cb0c5830836441781dbf41da1e79569 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240cb0c5830836441781dbf41da1e79569 2023-03-24 21:56:15,524 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,527 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,529 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,531 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,532 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,532 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,532 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,532 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,534 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,534 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6148eba4[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,535 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,536 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,536 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,537 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,537 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,541 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,541 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,541 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,542 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,544 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,544 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,544 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,544 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,545 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,546 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,546 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,547 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,549 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,550 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,550 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,550 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,650 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,653 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,654 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,655 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,655 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,657 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,657 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,657 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,657 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,657 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,660 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,660 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,661 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,661 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7907091d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,663 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,664 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,664 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,664 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,665 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,668 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,668 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,668 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,669 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,669 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,670 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,670 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,671 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,674 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,675 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,675 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,675 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,675 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,719 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695035718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,775 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,777 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,779 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,779 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,783 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,783 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,785 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,785 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,787 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,787 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eeed7f5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,789 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,790 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,791 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,791 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,791 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,791 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,793 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,796 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,796 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,796 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,798 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,798 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,800 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,802 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,802 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,802 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,802 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,903 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:15,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:15,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:15,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:15,906 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:15,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:15,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:15,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,909 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,909 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:15,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,910 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:15,910 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:15,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:15,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:15,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:15,910 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:15,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:15,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:15,911 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:15,911 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:15,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,913 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:15,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e7d7d6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:15,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:15,914 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,915 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:15,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:15,915 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:15,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,916 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:15,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695035912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,916 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=148, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/671c0094245149eeb3a2449d0148e53f 2023-03-24 21:56:15,918 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:15,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,918 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:15,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695035918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:15,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:15,922 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:15,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:15,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:15,922 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:15,922 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:15,930 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,930 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,932 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:15,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:15,933 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,934 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:15,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,936 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/f673a5fe0e0540c59a37bfe2ea62cf75 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,937 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:15,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:15,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:15,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:15,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:15,937 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:15,937 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:15,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:15,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:15,937 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:15,946 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/d14ce7ad78214054a8726353018c14a5 2023-03-24 21:56:15,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/671c0094245149eeb3a2449d0148e53f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f 2023-03-24 21:56:15,953 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f, entries=150, sequenceid=148, filesize=25.3 K 2023-03-24 21:56:15,953 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/f673a5fe0e0540c59a37bfe2ea62cf75 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75 2023-03-24 21:56:15,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75, entries=150, sequenceid=148, filesize=11.8 K 2023-03-24 21:56:15,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/d14ce7ad78214054a8726353018c14a5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5 2023-03-24 21:56:15,961 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5, entries=150, sequenceid=148, filesize=11.8 K 2023-03-24 21:56:15,963 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 09571a8f4d95ff573f25f32562564d3a in 881ms, sequenceid=148, compaction requested=true 2023-03-24 21:56:15,963 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:15,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:15,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:56:15,963 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:56:15,963 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:15,963 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:15,965 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103237 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:15,965 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:15,965 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,965 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7878085a61ad4b01bdcfc9debb890689, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=100.8 K 2023-03-24 21:56:15,965 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:15,965 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:15,965 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:15,965 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a98898c77722416cbf39d509cb9b9486, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.7 K 2023-03-24 21:56:15,966 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7878085a61ad4b01bdcfc9debb890689, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972960 2023-03-24 21:56:15,966 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 0595d22b4e47443591fd54df4eec01be, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694973013 2023-03-24 21:56:15,966 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a98898c77722416cbf39d509cb9b9486, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972960 2023-03-24 21:56:15,966 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c4e0216dd0cf49f39fb902a65956fec4, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679694973624 2023-03-24 21:56:15,967 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 448e38d200954632aaf17c2c645b7e8f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694973013 2023-03-24 21:56:15,967 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 671c0094245149eeb3a2449d0148e53f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:15,967 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0893d7e943214e548ccd6e27841ee654, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679694973624 2023-03-24 21:56:15,967 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f673a5fe0e0540c59a37bfe2ea62cf75, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:15,995 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#527 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:15,998 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#528 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:16,038 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:16,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:16,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:16,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:16,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:16,041 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:16,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:16,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:16,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:16,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,041 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,042 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,042 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:16,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:16,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:16,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,043 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:16,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:16,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:16,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:16,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:16,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:16,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:16,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:16,044 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:16,044 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:16,044 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:16,063 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324730e451ac18f4565afedeba45af34565 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324730e451ac18f4565afedeba45af34565 2023-03-24 21:56:16,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:16,225 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:16,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694976270, max timestamp=1679694976281Num uniques 859; 2023-03-24 21:56:16,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694976270, max timestamp=1679694976281Num uniques 859; 2023-03-24 21:56:16,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694976270, max timestamp=1679694976281Num uniques 859; 2023-03-24 21:56:16,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:16,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:16,284 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:16,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695036337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,411 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/cbd466cd58374e02bc62d4708608bbb1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cbd466cd58374e02bc62d4708608bbb1 2023-03-24 21:56:16,415 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into cbd466cd58374e02bc62d4708608bbb1(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:16,415 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:16,415 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694975963; duration=0sec 2023-03-24 21:56:16,415 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:16,415 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:16,416 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47797 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:16,417 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:16,417 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:16,417 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f33f95cda54e43c4896c8a635414f18f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.7 K 2023-03-24 21:56:16,417 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f33f95cda54e43c4896c8a635414f18f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1679694972960 2023-03-24 21:56:16,417 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d5daebdc8d8a47ebae7446771c17a960, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679694973013 2023-03-24 21:56:16,418 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 03ea24a9664149df832e03406a3ea3bc, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1679694973624 2023-03-24 21:56:16,418 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d14ce7ad78214054a8726353018c14a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:16,426 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#530 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:16,430 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/d82ef0ca17c540269ecfb6b62c17b4eb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d82ef0ca17c540269ecfb6b62c17b4eb 2023-03-24 21:56:16,434 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into d82ef0ca17c540269ecfb6b62c17b4eb(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:16,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:16,434 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=12, startTime=1679694975963; duration=0sec 2023-03-24 21:56:16,434 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:16,436 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9fccf2c0bf9b4571a86cd155487fe32c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9fccf2c0bf9b4571a86cd155487fe32c 2023-03-24 21:56:16,440 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 9fccf2c0bf9b4571a86cd155487fe32c(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:16,440 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:16,440 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694975963; duration=0sec 2023-03-24 21:56:16,440 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:16,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695036441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,469 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/6977d6955ada41b7861e0b0cbf37be67 2023-03-24 21:56:16,478 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/1c6b4e2d8ee74e1186f6d01dfdf0b516 2023-03-24 21:56:16,487 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/21f6eff35eb44597b9201da41dcd8f62 2023-03-24 21:56:16,491 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/6977d6955ada41b7861e0b0cbf37be67 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67 2023-03-24 21:56:16,494 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67, entries=150, sequenceid=158, filesize=25.3 K 2023-03-24 21:56:16,495 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/1c6b4e2d8ee74e1186f6d01dfdf0b516 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516 2023-03-24 21:56:16,498 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516, entries=150, sequenceid=158, filesize=11.8 K 2023-03-24 21:56:16,498 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/21f6eff35eb44597b9201da41dcd8f62 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62 2023-03-24 21:56:16,502 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62, entries=150, sequenceid=158, filesize=11.8 K 2023-03-24 21:56:16,503 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 09571a8f4d95ff573f25f32562564d3a in 459ms, sequenceid=158, compaction requested=false 2023-03-24 21:56:16,503 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:16,503 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:16,503 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:16,503 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:16,503 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,503 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:16,503 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:16,505 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:16,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:16,505 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:16,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:16,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:16,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62540125[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:16,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:16,507 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,510 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:16,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:16,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:16,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:16,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:16,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:16,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:16,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:16,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:16,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:16,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:16,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:16,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:16,517 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:16,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:16,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:16,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:16,518 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:16,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:16,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:16,520 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:16,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:16,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:16,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:16,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:16,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:16,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:16,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:16,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,524 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:16,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:16,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:16,524 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:16,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:16,625 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:16,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:16,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:16,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:16,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:16,629 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:16,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:16,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:16,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:16,629 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,636 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:16,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:16,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:16,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:16,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:16,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:16,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:16,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:16,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:16,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:16,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:16,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:16,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:16,638 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:16,638 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:16,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:16,648 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:16,651 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a73f838aa6814f9f8bd32242a90609be to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a73f838aa6814f9f8bd32242a90609be 2023-03-24 21:56:16,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695036671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695036775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695036921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695036924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695036978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,056 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/129fae510c6b4ae388ab45ee1001a22b 2023-03-24 21:56:17,067 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/146cc0ff4565495e9cdc7d1740e13086 2023-03-24 21:56:17,079 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/524686bcab4741259ee9f358697605f7 2023-03-24 21:56:17,082 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/129fae510c6b4ae388ab45ee1001a22b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b 2023-03-24 21:56:17,086 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b, entries=150, sequenceid=198, filesize=25.3 K 2023-03-24 21:56:17,087 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/146cc0ff4565495e9cdc7d1740e13086 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086 2023-03-24 21:56:17,092 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:56:17,093 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/524686bcab4741259ee9f358697605f7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7 2023-03-24 21:56:17,097 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7, entries=150, sequenceid=198, filesize=11.8 K 2023-03-24 21:56:17,097 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 09571a8f4d95ff573f25f32562564d3a in 459ms, sequenceid=198, compaction requested=true 2023-03-24 21:56:17,097 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:17,097 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,097 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,098 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,099 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,100 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,100 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23dab393[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,101 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,105 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,105 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,105 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,107 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,110 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,110 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,112 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,117 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,117 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:17,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:17,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:56:17,160 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:17,161 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:17,217 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694977209, max timestamp=1679694977221Num uniques 859; 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694977209, max timestamp=1679694977221Num uniques 859; 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694977209, max timestamp=1679694977221Num uniques 859; 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:17,223 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:17,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,227 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,227 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,228 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,229 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,229 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,230 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,231 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,231 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,248 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,248 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,248 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,249 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,249 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,252 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bd39962[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,256 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,256 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,256 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,257 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,257 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,257 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,257 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,262 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,262 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,262 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,262 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,262 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,264 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,266 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695037264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,267 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695037265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,271 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,272 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,272 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,272 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695037283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,369 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695037368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695037369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,372 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,376 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,376 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,379 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,380 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,380 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,380 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,380 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,382 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,382 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,383 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f0a8a9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,386 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,386 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,386 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,388 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,392 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,392 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,392 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,394 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,398 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,401 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,401 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,401 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:17,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,402 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,402 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,502 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,506 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,508 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,509 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,509 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,509 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,509 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,510 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,510 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,510 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,510 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,514 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,514 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43efdc9b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,517 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,522 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,524 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,524 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,530 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,532 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,532 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,533 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,533 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,533 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695037571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695037571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,612 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303246049e4f7640d4c4fa97e9cf507ea1c4e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246049e4f7640d4c4fa97e9cf507ea1c4e 2023-03-24 21:56:17,633 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,636 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,636 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,638 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,638 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,638 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,639 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,639 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,639 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,641 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,641 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@355fb95e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,643 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,643 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,645 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,645 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,645 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,645 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,645 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,645 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,647 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,647 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,650 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,650 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,651 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,651 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,651 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,652 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,656 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:17,656 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,656 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,656 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,756 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,760 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,760 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,761 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,765 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,765 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,765 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,765 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,765 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,765 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,765 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,767 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,767 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,767 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,767 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52a03e1e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,771 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,771 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,771 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,772 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,775 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,775 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,776 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,781 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,781 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,781 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,782 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,784 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,786 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,786 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,787 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,787 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56590 deadline: 1679695037789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695037874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:17,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695037876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,887 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:17,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:17,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:17,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:17,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:17,890 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:17,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:17,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:17,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,891 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,892 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:17,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:17,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:17,893 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,893 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:17,893 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:17,893 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:17,893 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:17,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,894 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,895 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:17,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56916103[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:17,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:17,896 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,899 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:17,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:17,899 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:17,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,899 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:17,900 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,900 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:17,900 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,900 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:17,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:17,903 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:17,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:17,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:17,903 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:17,903 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:17,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:17,904 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:17,904 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,906 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:17,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:17,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,913 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:17,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:17,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:17,913 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:17,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:17,914 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:17,914 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:17,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,014 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:18,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:18,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:18,017 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/0be26f3acbdd44079d5bd03387d6aeb1 2023-03-24 21:56:18,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:18,017 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:18,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:18,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:18,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,018 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,019 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,019 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,020 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:18,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:18,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:18,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:18,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:18,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:18,021 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:18,021 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,021 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:18,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:18,022 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:18,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:18,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:18,022 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:18,023 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,023 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,024 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:18,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ba7744e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:18,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:18,025 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,026 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,026 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:18,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:18,027 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:18,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,027 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:18,030 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:18,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,030 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:18,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:18,032 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:18,032 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:18,032 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,034 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,035 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,037 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:18,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,039 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,039 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,039 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:18,040 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:18,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,140 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:18,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:18,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:18,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:18,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:18,143 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:18,143 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,145 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,146 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,146 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,146 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:18,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:18,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:18,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:18,147 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:18,147 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,147 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:18,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:18,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:18,148 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:18,148 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:18,151 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:18,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@237ab52d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:18,153 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:18,154 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,155 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:18,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:18,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:18,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,155 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,157 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,157 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:18,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:18,160 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:18,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:18,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:18,160 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:18,160 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:18,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,162 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,162 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,164 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,167 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:18,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:18,167 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,168 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:18,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,168 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:18,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:18,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,168 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,268 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:18,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:18,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:18,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:18,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:18,271 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,272 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,273 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,274 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:18,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:18,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:18,275 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,275 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:18,275 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:18,275 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:18,275 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:18,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:18,276 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:18,277 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,277 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:18,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:18,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70793f92[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:18,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:18,279 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,280 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:18,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:18,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,280 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:18,280 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:18,283 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,283 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:18,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:18,285 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:18,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:18,285 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:18,286 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:18,286 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:18,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,287 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,288 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,289 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,296 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:18,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,296 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,296 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,296 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56540 deadline: 1679695038378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56554 deadline: 1679695038383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,396 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:18,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:18,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:18,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:18,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:18,399 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:18,399 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:18,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:18,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,402 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,402 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,403 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:18,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:18,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:18,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:18,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:18,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:18,404 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:18,404 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:18,405 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,406 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,406 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:18,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:18,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dbdc1e7[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:18,408 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:18,409 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,410 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:18,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:18,410 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,410 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:18,411 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:18,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,411 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:18,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:18,414 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:18,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:18,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:18,414 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:18,414 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:18,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,416 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:18,417 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:18,418 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:18,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,422 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:18,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:18,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:18,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:18,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:18,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,422 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,422 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:18,422 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,428 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/0de3e1f53d4b4bd09fa2c0d367318fcd 2023-03-24 21:56:18,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fd7d84e207a24251b0fb58293517cc19 2023-03-24 21:56:18,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/0be26f3acbdd44079d5bd03387d6aeb1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1 2023-03-24 21:56:18,443 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1, entries=150, sequenceid=209, filesize=25.3 K 2023-03-24 21:56:18,443 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/0de3e1f53d4b4bd09fa2c0d367318fcd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd 2023-03-24 21:56:18,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd, entries=100, sequenceid=209, filesize=9.4 K 2023-03-24 21:56:18,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fd7d84e207a24251b0fb58293517cc19 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19 2023-03-24 21:56:18,452 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19, entries=100, sequenceid=209, filesize=9.4 K 2023-03-24 21:56:18,453 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09571a8f4d95ff573f25f32562564d3a in 1293ms, sequenceid=209, compaction requested=true 2023-03-24 21:56:18,453 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:18,453 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:18,453 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:18,454 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:18,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:18,454 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:18,455 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103820 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:18,455 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:18,455 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:18,455 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,455 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:18,455 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d82ef0ca17c540269ecfb6b62c17b4eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=101.4 K 2023-03-24 21:56:18,455 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,455 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cbd466cd58374e02bc62d4708608bbb1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=44.9 K 2023-03-24 21:56:18,455 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d82ef0ca17c540269ecfb6b62c17b4eb, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:18,456 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting cbd466cd58374e02bc62d4708608bbb1, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:18,456 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6977d6955ada41b7861e0b0cbf37be67, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679694975093 2023-03-24 21:56:18,456 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 1c6b4e2d8ee74e1186f6d01dfdf0b516, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679694975093 2023-03-24 21:56:18,457 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 129fae510c6b4ae388ab45ee1001a22b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694976325 2023-03-24 21:56:18,457 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 146cc0ff4565495e9cdc7d1740e13086, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694976325 2023-03-24 21:56:18,457 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 0be26f3acbdd44079d5bd03387d6aeb1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976661 2023-03-24 21:56:18,457 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0de3e1f53d4b4bd09fa2c0d367318fcd, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976661 2023-03-24 21:56:18,465 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:18,467 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#539 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:18,480 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/e27a5d800e2248368a739319c1b31e69 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/e27a5d800e2248368a739319c1b31e69 2023-03-24 21:56:18,487 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into e27a5d800e2248368a739319c1b31e69(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:18,487 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:18,487 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694978453; duration=0sec 2023-03-24 21:56:18,487 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:18,487 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:18,488 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45990 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:18,488 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:18,489 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,489 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9fccf2c0bf9b4571a86cd155487fe32c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=44.9 K 2023-03-24 21:56:18,489 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9fccf2c0bf9b4571a86cd155487fe32c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679694974790 2023-03-24 21:56:18,490 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 21f6eff35eb44597b9201da41dcd8f62, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679694975093 2023-03-24 21:56:18,490 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 524686bcab4741259ee9f358697605f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1679694976325 2023-03-24 21:56:18,491 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting fd7d84e207a24251b0fb58293517cc19, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976661 2023-03-24 21:56:18,499 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#541 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:18,504 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/19ae32ed760243fc9ae2b3e797df8f15 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/19ae32ed760243fc9ae2b3e797df8f15 2023-03-24 21:56:18,510 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 19ae32ed760243fc9ae2b3e797df8f15(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:18,510 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:18,510 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=12, startTime=1679694978453; duration=0sec 2023-03-24 21:56:18,510 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:18,515 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/c95b69031c444f018e752eb40b4ee6b3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c95b69031c444f018e752eb40b4ee6b3 2023-03-24 21:56:18,519 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into c95b69031c444f018e752eb40b4ee6b3(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:18,519 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:18,519 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694978454; duration=0sec 2023-03-24 21:56:18,519 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:18,523 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:18,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:18,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:18,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:18,526 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:18,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:18,526 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:18,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:18,528 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:18,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,529 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:18,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:18,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:18,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:18,530 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:18,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:18,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:18,530 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 21:56:18,530 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:18,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:18,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:18,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:18,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:18,531 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:18,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:18,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:18,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695038816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,919 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695038918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,926 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695038924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,928 DEBUG [Thread-3883] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:56:18,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:18,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695038942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:18,944 DEBUG [Thread-3887] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:56:18,953 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324901cfbf548dd4381b822801d6a7cb2e5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324901cfbf548dd4381b822801d6a7cb2e5 2023-03-24 21:56:18,959 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=248, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/59f6d11d945c477d961e498a56ab406b 2023-03-24 21:56:18,975 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/797d28cf3f73496dbeed46964faf0d80 2023-03-24 21:56:18,986 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/44b8b99dc5df4c418a94ffaa4202e53c 2023-03-24 21:56:18,990 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/59f6d11d945c477d961e498a56ab406b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b 2023-03-24 21:56:18,993 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b, entries=150, sequenceid=248, filesize=25.3 K 2023-03-24 21:56:18,994 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/797d28cf3f73496dbeed46964faf0d80 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80 2023-03-24 21:56:18,997 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80, entries=150, sequenceid=248, filesize=11.8 K 2023-03-24 21:56:18,998 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/44b8b99dc5df4c418a94ffaa4202e53c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c 2023-03-24 21:56:19,001 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c, entries=150, sequenceid=248, filesize=11.8 K 2023-03-24 21:56:19,002 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 09571a8f4d95ff573f25f32562564d3a in 472ms, sequenceid=248, compaction requested=false 2023-03-24 21:56:19,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d68d51a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,007 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,009 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,009 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,010 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,013 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,013 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,013 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,013 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,014 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,015 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,016 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,020 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,021 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,021 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,022 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,022 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,122 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:19,122 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:19,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:19,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,139 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,139 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,140 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,142 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,151 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,152 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,152 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,154 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,154 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,155 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bcdd447[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,157 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,159 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,159 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,159 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,159 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,160 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,161 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,162 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324779e4b81646d406ea22555d3c97bbe21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324779e4b81646d406ea22555d3c97bbe21 2023-03-24 21:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,171 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,171 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,171 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,174 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,176 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,178 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=260, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/7ff32124fd85489e827d46961e46fa64 2023-03-24 21:56:19,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,179 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,184 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695039182, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,185 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,185 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,201 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/20f61cf4e39445649e17f19f5c3ac8a0 2023-03-24 21:56:19,215 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=260 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fb90e4ac6a21429e90b9d7b7f3bf10c5 2023-03-24 21:56:19,221 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/7ff32124fd85489e827d46961e46fa64 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64 2023-03-24 21:56:19,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64, entries=150, sequenceid=260, filesize=25.3 K 2023-03-24 21:56:19,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/20f61cf4e39445649e17f19f5c3ac8a0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0 2023-03-24 21:56:19,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0, entries=150, sequenceid=260, filesize=11.8 K 2023-03-24 21:56:19,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fb90e4ac6a21429e90b9d7b7f3bf10c5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5 2023-03-24 21:56:19,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5, entries=150, sequenceid=260, filesize=11.8 K 2023-03-24 21:56:19,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 09571a8f4d95ff573f25f32562564d3a in 112ms, sequenceid=260, compaction requested=true 2023-03-24 21:56:19,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:19,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:19,234 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:19,234 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:19,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:19,236 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36534 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:19,236 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:19,236 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,237 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/e27a5d800e2248368a739319c1b31e69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=35.7 K 2023-03-24 21:56:19,237 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78114 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:19,237 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:19,237 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,237 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/19ae32ed760243fc9ae2b3e797df8f15, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=76.3 K 2023-03-24 21:56:19,237 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e27a5d800e2248368a739319c1b31e69, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976325 2023-03-24 21:56:19,238 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 797d28cf3f73496dbeed46964faf0d80, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679694977258 2023-03-24 21:56:19,238 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 20f61cf4e39445649e17f19f5c3ac8a0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:19,238 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 19ae32ed760243fc9ae2b3e797df8f15, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976325 2023-03-24 21:56:19,239 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 59f6d11d945c477d961e498a56ab406b, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679694977258 2023-03-24 21:56:19,240 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7ff32124fd85489e827d46961e46fa64, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:19,246 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#548 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:19,256 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#549 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:19,272 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/a8e6397776f54685870eb0d2301d9abe as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a8e6397776f54685870eb0d2301d9abe 2023-03-24 21:56:19,276 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into a8e6397776f54685870eb0d2301d9abe(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:19,276 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,276 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694979234; duration=0sec 2023-03-24 21:56:19,277 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:19,277 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:19,278 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36534 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:19,278 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:19,279 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,279 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c95b69031c444f018e752eb40b4ee6b3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=35.7 K 2023-03-24 21:56:19,279 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting c95b69031c444f018e752eb40b4ee6b3, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679694976325 2023-03-24 21:56:19,279 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 44b8b99dc5df4c418a94ffaa4202e53c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679694977258 2023-03-24 21:56:19,280 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting fb90e4ac6a21429e90b9d7b7f3bf10c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:19,285 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,286 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#550 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:19,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,290 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,290 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,291 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:19,295 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979144, max timestamp=1679694979286Num uniques 859; 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979144, max timestamp=1679694979286Num uniques 859; 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979144, max timestamp=1679694979286Num uniques 859; 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,297 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,296 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,296 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-03-24 21:56:19,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:19,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:19,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:19,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,298 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,298 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,299 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9960355b03534b3e9ce8e875f90fbcba as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9960355b03534b3e9ce8e875f90fbcba 2023-03-24 21:56:19,302 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,302 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,303 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,303 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/ee1273871c9941d79f7e3b1be4f7c6b0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ee1273871c9941d79f7e3b1be4f7c6b0 2023-03-24 21:56:19,305 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 9960355b03534b3e9ce8e875f90fbcba(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:19,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,306 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,306 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,306 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,306 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694979234; duration=0sec 2023-03-24 21:56:19,306 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,306 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,306 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:19,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,309 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into ee1273871c9941d79f7e3b1be4f7c6b0(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:19,309 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,309 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694979234; duration=0sec 2023-03-24 21:56:19,309 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:19,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,310 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3271ff03[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,313 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,314 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,315 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,316 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,319 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,320 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,322 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,324 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,327 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,327 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,327 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,328 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303245261d6e4ee8548b883f18949d2b405ec to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245261d6e4ee8548b883f18949d2b405ec 2023-03-24 21:56:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,331 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,331 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,358 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=282, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/5ffbd7a7d44e4892b58e391918cdcdc3 2023-03-24 21:56:19,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979353, max timestamp=1679694979368Num uniques 859; 2023-03-24 21:56:19,370 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979353, max timestamp=1679694979368Num uniques 859; 2023-03-24 21:56:19,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694979353, max timestamp=1679694979368Num uniques 859; 2023-03-24 21:56:19,374 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:19,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694979392, max timestamp=1679694979402Num uniques 1018; 2023-03-24 21:56:19,404 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:19,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694979392, max timestamp=1679694979402Num uniques 1018; 2023-03-24 21:56:19,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:19,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694979392, max timestamp=1679694979402Num uniques 1018; 2023-03-24 21:56:19,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:19,431 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,434 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,442 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,443 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7180b253[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,446 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,447 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1679694979417, max timestamp=1679694979438Num uniques 1027; 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1679694979417, max timestamp=1679694979438Num uniques 1027; 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10382, totalHeapSize=23848, min timestamp=1679694979417, max timestamp=1679694979438Num uniques 1027; 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:19,452 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=10382, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:19,455 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,455 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,455 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,455 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,459 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,459 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,459 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,461 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,463 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,466 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,466 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679694979464, max timestamp=1679694979484Num uniques 1086; 2023-03-24 21:56:19,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-24 21:56:19,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694979464, max timestamp=1679694979484Num uniques 1086; 2023-03-24 21:56:19,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-03-24 21:56:19,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694979464, max timestamp=1679694979484Num uniques 1086; 2023-03-24 21:56:19,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=6870, suffix heap size=140912,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=150 2023-03-24 21:56:19,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694979504, max timestamp=1679694979515Num uniques 1009; 2023-03-24 21:56:19,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-24 21:56:19,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694979504, max timestamp=1679694979515Num uniques 1095; 2023-03-24 21:56:19,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679694979504, max timestamp=1679694979515Num uniques 1009; 2023-03-24 21:56:19,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-24 21:56:19,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8930, suffix heap size=132904,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=195 2023-03-24 21:56:19,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694979535, max timestamp=1679694979541Num uniques 1054; 2023-03-24 21:56:19,543 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694979535, max timestamp=1679694979541Num uniques 1054; 2023-03-24 21:56:19,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:56:19,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:56:19,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679694979535, max timestamp=1679694979541Num uniques 1054; 2023-03-24 21:56:19,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-24 21:56:19,567 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679694979559, max timestamp=1679694979566Num uniques 1013; 2023-03-24 21:56:19,568 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:56:19,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7455, totalHeapSize=17192, min timestamp=1679694979559, max timestamp=1679694979566Num uniques 1013; 2023-03-24 21:56:19,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7456, totalHeapSize=17192, min timestamp=1679694979559, max timestamp=1679694979566Num uniques 1013; 2023-03-24 21:56:19,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46385, new segment data size=7455, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:56:19,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46386, new segment data size=7456, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-24 21:56:19,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,570 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,571 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,573 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,574 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,575 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,575 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,575 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,575 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,579 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a7b797c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,582 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,583 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,584 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,584 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,585 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,588 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,588 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,590 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,592 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,595 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,595 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,596 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,596 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,596 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,600 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679694979581, max timestamp=1679694979599Num uniques 1022; 2023-03-24 21:56:19,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-24 21:56:19,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679694979581, max timestamp=1679694979599Num uniques 1022; 2023-03-24 21:56:19,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-24 21:56:19,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679694979581, max timestamp=1679694979599Num uniques 1022; 2023-03-24 21:56:19,601 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-24 21:56:19,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694979614, max timestamp=1679694979622Num uniques 1131; 2023-03-24 21:56:19,624 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-24 21:56:19,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694979614, max timestamp=1679694979622Num uniques 1131; 2023-03-24 21:56:19,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1679694979614, max timestamp=1679694979622Num uniques 1131; 2023-03-24 21:56:19,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-24 21:56:19,625 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=10576, suffix heap size=145592,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=231 2023-03-24 21:56:19,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679694979638, max timestamp=1679694979643Num uniques 1090; 2023-03-24 21:56:19,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-24 21:56:19,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1679694979638, max timestamp=1679694979643Num uniques 1090; 2023-03-24 21:56:19,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1679694979640, max timestamp=1679694979643Num uniques 1090; 2023-03-24 21:56:19,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=13280, suffix heap size=141328,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=290 2023-03-24 21:56:19,646 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=8700, suffix heap size=141328,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=190 2023-03-24 21:56:19,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679694979683, max timestamp=1679694979688Num uniques 1149; 2023-03-24 21:56:19,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-24 21:56:19,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679694979683, max timestamp=1679694979688Num uniques 1149; 2023-03-24 21:56:19,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=9114, suffix heap size=147464,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=199 2023-03-24 21:56:19,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1679694979683, max timestamp=1679694979688Num uniques 1049; 2023-03-24 21:56:19,691 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-03-24 21:56:19,696 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,699 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,700 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,701 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,703 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,704 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,708 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,709 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f71f398[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,711 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,711 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,714 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679694979703, max timestamp=1679694979712Num uniques 1058; 2023-03-24 21:56:19,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1679694979703, max timestamp=1679694979712Num uniques 1058; 2023-03-24 21:56:19,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-24 21:56:19,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-03-24 21:56:19,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,715 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679694979704, max timestamp=1679694979712Num uniques 1058; 2023-03-24 21:56:19,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=6870, suffix heap size=138000,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=150 2023-03-24 21:56:19,716 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,717 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,717 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,718 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,718 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,718 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,725 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,725 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,725 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,725 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,725 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679694979727, max timestamp=1679694979733Num uniques 1067; 2023-03-24 21:56:19,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-03-24 21:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1679694979727, max timestamp=1679694979733Num uniques 1067; 2023-03-24 21:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-03-24 21:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1679694979729, max timestamp=1679694979733Num uniques 1009; 2023-03-24 21:56:19,736 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=46212, new segment data size=7642, suffix heap size=132904,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=167 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1679694979749, max timestamp=1679694979753Num uniques 1026; 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=8056, suffix heap size=134672,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=176 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8046, totalHeapSize=18544, min timestamp=1679694979749, max timestamp=1679694979753Num uniques 1126; 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8055, totalHeapSize=18544, min timestamp=1679694979749, max timestamp=1679694979753Num uniques 1126; 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=51565, new segment data size=8055, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-03-24 21:56:19,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=51556, new segment data size=8046, suffix heap size=145072,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1126, new segment cells count=176 2023-03-24 21:56:19,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/7790a2ff2cbc4c6ab49037820ab675c2 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679694979769, max timestamp=1679694979780Num uniques 1035; 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679694979769, max timestamp=1679694979780Num uniques 1035; 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=285, cellSize=13050, totalHeapSize=29880, min timestamp=1679694979769, max timestamp=1679694979780Num uniques 1035; 2023-03-24 21:56:19,785 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=13050, suffix heap size=135608,new segment heap size=29880 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=285 2023-03-24 21:56:19,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/4d4bfc73ad22494a859817d0571258d3 2023-03-24 21:56:19,806 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/5ffbd7a7d44e4892b58e391918cdcdc3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3 2023-03-24 21:56:19,812 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3, entries=150, sequenceid=282, filesize=25.4 K 2023-03-24 21:56:19,813 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/7790a2ff2cbc4c6ab49037820ab675c2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2 2023-03-24 21:56:19,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679694979809, max timestamp=1679694979814Num uniques 1144; 2023-03-24 21:56:19,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-03-24 21:56:19,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679694979809, max timestamp=1679694979814Num uniques 1144; 2023-03-24 21:56:19,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-03-24 21:56:19,817 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2, entries=150, sequenceid=282, filesize=11.9 K 2023-03-24 21:56:19,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1679694979809, max timestamp=1679694979814Num uniques 1144; 2023-03-24 21:56:19,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=52394, new segment data size=8884, suffix heap size=146944,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1144, new segment cells count=194 2023-03-24 21:56:19,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/4d4bfc73ad22494a859817d0571258d3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3 2023-03-24 21:56:19,824 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3, entries=150, sequenceid=282, filesize=11.9 K 2023-03-24 21:56:19,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~66.19 KB/67776, currentSize=40.25 KB/41220 for 09571a8f4d95ff573f25f32562564d3a in 529ms, sequenceid=282, compaction requested=false 2023-03-24 21:56:19,825 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:19,825 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:19,827 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=142.64 KB 2023-03-24 21:56:19,828 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:19,828 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,828 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:19,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,828 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,829 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:19,829 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,833 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,833 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,834 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,835 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,835 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,837 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,843 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,844 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,844 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,844 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,845 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,846 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,846 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,847 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12459892[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,849 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,850 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,850 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,850 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,851 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,851 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,856 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,856 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694979849, max timestamp=1679694979856Num uniques 859; 2023-03-24 21:56:19,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:19,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694979849, max timestamp=1679694979856Num uniques 859; 2023-03-24 21:56:19,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694979849, max timestamp=1679694979856Num uniques 859; 2023-03-24 21:56:19,858 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,861 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,861 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,861 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,864 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,864 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,865 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242c49156114b04bf6b2e571a339d3277a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c49156114b04bf6b2e571a339d3277a 2023-03-24 21:56:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,866 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,869 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,869 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,869 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,869 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,869 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:19,869 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,869 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,869 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695039878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695039877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695039881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,969 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:19,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:19,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:19,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:19,973 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:19,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:19,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:19,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,975 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:19,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:19,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:19,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:19,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:19,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:19,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:19,976 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:19,976 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:19,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:19,977 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:19,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:19,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:19,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:19,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:19,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,980 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,981 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6623b2a3[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:19,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:19,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:19,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:19,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695039982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,984 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:19,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695039982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,985 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695039984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,985 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:19,986 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:19,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:19,989 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:19,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:19,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:19,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:19,989 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:19,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:19,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,991 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:19,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,993 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:19,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:19,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:19,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:19,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:19,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:19,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:19,999 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:19,999 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:20,099 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:20,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:20,102 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:20,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,102 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:20,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:20,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,104 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:20,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:20,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,105 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:20,106 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:20,106 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:20,106 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,106 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:20,106 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:20,106 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:20,106 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:20,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,108 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77028782[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:20,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:20,110 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,111 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:20,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:20,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:20,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,111 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:20,117 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:20,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,117 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:20,119 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:20,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:20,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:20,119 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:20,119 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:20,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,121 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:20,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:20,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,123 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:20,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:20,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:20,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:20,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,126 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,125 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:20,125 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:20,126 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:20,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:20,126 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:20,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695040186, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695040187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695040187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,226 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:20,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:20,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:20,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:20,230 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:20,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:20,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:20,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,230 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,231 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:20,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:20,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,232 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,233 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:20,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:20,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:20,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:20,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:20,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:20,233 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,234 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:20,234 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:20,234 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,234 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:20,234 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:20,234 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,234 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:20,234 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:20,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,236 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@574f57c9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:20,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:20,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:20,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:20,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:20,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:20,240 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:20,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:20,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:20,243 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:20,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:20,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:20,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:20,243 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:20,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:20,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,245 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:20,245 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:20,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:20,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:20,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:20,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:20,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:20,251 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:20,251 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:20,251 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:20,272 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=567, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/b920aba130c94e1688a088c1cb5878ea 2023-03-24 21:56:20,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8745db128ed7498a92269d320e34209a 2023-03-24 21:56:20,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=567 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/295b2f5a00e644f294965d457a394e5f 2023-03-24 21:56:20,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/b920aba130c94e1688a088c1cb5878ea as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea 2023-03-24 21:56:20,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea, entries=150, sequenceid=567, filesize=25.4 K 2023-03-24 21:56:20,302 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8745db128ed7498a92269d320e34209a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a 2023-03-24 21:56:20,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a, entries=150, sequenceid=567, filesize=11.9 K 2023-03-24 21:56:20,306 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/295b2f5a00e644f294965d457a394e5f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f 2023-03-24 21:56:20,313 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f, entries=150, sequenceid=567, filesize=11.9 K 2023-03-24 21:56:20,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~162.52 KB/166416, currentSize=127.47 KB/130530 for 09571a8f4d95ff573f25f32562564d3a in 487ms, sequenceid=567, compaction requested=true 2023-03-24 21:56:20,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:20,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:20,314 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:20,314 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:20,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:20,315 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:20,316 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78516 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:20,316 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:20,316 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,316 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ee1273871c9941d79f7e3b1be4f7c6b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=76.7 K 2023-03-24 21:56:20,316 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ee1273871c9941d79f7e3b1be4f7c6b0, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:20,317 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:20,317 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5ffbd7a7d44e4892b58e391918cdcdc3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679694979144 2023-03-24 21:56:20,317 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:20,317 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,317 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a8e6397776f54685870eb0d2301d9abe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.1 K 2023-03-24 21:56:20,318 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b920aba130c94e1688a088c1cb5878ea, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:20,318 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting a8e6397776f54685870eb0d2301d9abe, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:20,319 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7790a2ff2cbc4c6ab49037820ab675c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679694979144 2023-03-24 21:56:20,319 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8745db128ed7498a92269d320e34209a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:20,326 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#558 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:20,332 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#557 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:20,346 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/31ee5e148ab94745a0c90c1bcfab3129 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/31ee5e148ab94745a0c90c1bcfab3129 2023-03-24 21:56:20,349 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into 31ee5e148ab94745a0c90c1bcfab3129(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:20,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:20,350 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694980314; duration=0sec 2023-03-24 21:56:20,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:20,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:20,350 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36936 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:20,351 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:20,351 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,351 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9960355b03534b3e9ce8e875f90fbcba, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.1 K 2023-03-24 21:56:20,351 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:20,351 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9960355b03534b3e9ce8e875f90fbcba, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=260, earliestPutTs=1679694978801 2023-03-24 21:56:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:20,352 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4d4bfc73ad22494a859817d0571258d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679694979144 2023-03-24 21:56:20,352 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 295b2f5a00e644f294965d457a394e5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:20,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:20,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:20,355 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:20,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:20,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:20,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,356 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,357 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,357 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:20,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:20,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,358 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,358 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:20,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:20,358 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#559 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:20,358 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:20,359 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:20,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:20,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:20,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:20,359 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:20,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:20,360 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:20,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:20,392 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324579a606aa841406b8c934832b9fe8180 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324579a606aa841406b8c934832b9fe8180 2023-03-24 21:56:20,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:20,492 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695040500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695040500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695040501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695040604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695040604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695040604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,777 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/aea33f7ccd504a3e8541d043e316d32c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/aea33f7ccd504a3e8541d043e316d32c 2023-03-24 21:56:20,781 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into aea33f7ccd504a3e8541d043e316d32c(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:20,781 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:20,781 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694980315; duration=0sec 2023-03-24 21:56:20,781 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:20,802 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=603, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/dbe8cbbf4a454bef8153e124f1073494 2023-03-24 21:56:20,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695040808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695040809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:20,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695040809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:20,814 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/320ac414ac404a3e8eb7afbdf825e2a5 2023-03-24 21:56:21,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695041114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695041115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695041115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,166 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/1d0bd61976a844d1bd019073ad03b6b9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d0bd61976a844d1bd019073ad03b6b9 2023-03-24 21:56:21,169 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 1d0bd61976a844d1bd019073ad03b6b9(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:21,170 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,170 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694980314; duration=0sec 2023-03-24 21:56:21,170 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:21,225 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=603 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5336c224592c4e7e894e390f192bddff 2023-03-24 21:56:21,229 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/dbe8cbbf4a454bef8153e124f1073494 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494 2023-03-24 21:56:21,232 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494, entries=150, sequenceid=603, filesize=25.4 K 2023-03-24 21:56:21,233 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/320ac414ac404a3e8eb7afbdf825e2a5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5 2023-03-24 21:56:21,236 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5, entries=150, sequenceid=603, filesize=11.9 K 2023-03-24 21:56:21,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5336c224592c4e7e894e390f192bddff as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff 2023-03-24 21:56:21,240 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff, entries=150, sequenceid=603, filesize=11.9 K 2023-03-24 21:56:21,240 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 09571a8f4d95ff573f25f32562564d3a in 880ms, sequenceid=603, compaction requested=false 2023-03-24 21:56:21,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,240 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,241 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,242 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6970c4ba[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,244 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,244 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,245 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,246 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,246 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,248 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,252 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,252 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,252 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,253 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,255 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,260 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:21,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,260 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,260 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,260 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,361 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:21,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:21,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:21,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:21,365 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:21,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:21,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:21,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,366 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,367 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,367 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,367 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:21,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,368 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:21,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:21,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:21,369 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:21,369 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,369 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:21,369 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:21,369 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:21,369 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:21,369 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:21,370 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,386 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324fd8cf31ab18147c1a2178619a51b0966 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fd8cf31ab18147c1a2178619a51b0966 2023-03-24 21:56:21,392 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=616, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/1d7ab8cf43684f74b41cea933a373ae4 2023-03-24 21:56:21,416 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/b4ebb9bc92094c11a90aecceadd012b3 2023-03-24 21:56:21,426 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/e781ba46430d4a54b4efbd19619dafcc 2023-03-24 21:56:21,431 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/1d7ab8cf43684f74b41cea933a373ae4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4 2023-03-24 21:56:21,435 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4, entries=150, sequenceid=616, filesize=25.4 K 2023-03-24 21:56:21,436 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/b4ebb9bc92094c11a90aecceadd012b3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3 2023-03-24 21:56:21,439 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3, entries=150, sequenceid=616, filesize=11.9 K 2023-03-24 21:56:21,440 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/e781ba46430d4a54b4efbd19619dafcc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc 2023-03-24 21:56:21,443 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc, entries=150, sequenceid=616, filesize=11.9 K 2023-03-24 21:56:21,444 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 75ms, sequenceid=616, compaction requested=true 2023-03-24 21:56:21,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,444 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,444 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,446 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,446 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@123dd59d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,448 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,449 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,450 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,451 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,454 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,454 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,460 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,461 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,463 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,465 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,466 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:21,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,466 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,466 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,466 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,567 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:21,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:21,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:21,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:21,570 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:21,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:21,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:21,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,572 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,572 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:21,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:21,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:21,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:21,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,573 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:21,574 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,574 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:21,574 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,574 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,574 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,576 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,576 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71606ecf[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,578 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,580 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,582 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,582 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,585 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,585 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,585 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,585 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,587 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,587 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,589 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,591 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:21,591 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,591 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,591 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,592 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,592 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,592 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,592 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,592 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,591 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:21,641 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:21,642 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:21,655 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244bce2eaa732a45abb294c8cb285bb124 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244bce2eaa732a45abb294c8cb285bb124 2023-03-24 21:56:21,661 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=627, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/52bc74c27f0e451787d2940bbb88af87 2023-03-24 21:56:21,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/6ea82f93f0be41edbbc8a134319406ef 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694981658, max timestamp=1679694981671Num uniques 859; 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694981658, max timestamp=1679694981671Num uniques 859; 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694981658, max timestamp=1679694981671Num uniques 859; 2023-03-24 21:56:21,673 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:21,678 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=627 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/b62ac4c32e974fb799d36fae4a9df12c 2023-03-24 21:56:21,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/52bc74c27f0e451787d2940bbb88af87 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87 2023-03-24 21:56:21,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87, entries=350, sequenceid=627, filesize=52.9 K 2023-03-24 21:56:21,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/6ea82f93f0be41edbbc8a134319406ef as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef 2023-03-24 21:56:21,688 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef, entries=150, sequenceid=627, filesize=11.9 K 2023-03-24 21:56:21,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/b62ac4c32e974fb799d36fae4a9df12c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c 2023-03-24 21:56:21,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c, entries=150, sequenceid=627, filesize=11.9 K 2023-03-24 21:56:21,692 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:21,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:21,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:21,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:21,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:21,695 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:21,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:21,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:21,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,697 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,697 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:21,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:21,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:21,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:21,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:21,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:21,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:21,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:21,699 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,701 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c3c9e35[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,703 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,703 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,705 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,705 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 09571a8f4d95ff573f25f32562564d3a in 64ms, sequenceid=627, compaction requested=true 2023-03-24 21:56:21,705 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:21,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:21,706 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:21,706 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:21,705 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:56:21,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,706 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:21,706 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=80.51 KB heapSize=220.92 KB 2023-03-24 21:56:21,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:21,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:21,706 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:21,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,706 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:21,707 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:21,707 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,708 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:21,708 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:21,708 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 132932 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:21,708 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,708 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:21,708 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/31ee5e148ab94745a0c90c1bcfab3129, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=48.2 K 2023-03-24 21:56:21,708 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,709 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d0bd61976a844d1bd019073ad03b6b9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=129.8 K 2023-03-24 21:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,709 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 31ee5e148ab94745a0c90c1bcfab3129, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:21,709 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,709 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,709 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,709 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1d0bd61976a844d1bd019073ad03b6b9, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:21,709 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 320ac414ac404a3e8eb7afbdf825e2a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1679694979864 2023-03-24 21:56:21,710 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting dbe8cbbf4a454bef8153e124f1073494, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1679694979864 2023-03-24 21:56:21,710 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b4ebb9bc92094c11a90aecceadd012b3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1679694980499 2023-03-24 21:56:21,710 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 1d7ab8cf43684f74b41cea933a373ae4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1679694980499 2023-03-24 21:56:21,710 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 6ea82f93f0be41edbbc8a134319406ef, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981634 2023-03-24 21:56:21,710 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 52bc74c27f0e451787d2940bbb88af87, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981633 2023-03-24 21:56:21,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,713 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,719 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,721 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,721 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,722 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:21,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,723 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,741 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#571 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:21,743 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244c78fef0b3a94afe822f1db1edf58830 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244c78fef0b3a94afe822f1db1edf58830 2023-03-24 21:56:21,756 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#570 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:21,759 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=655, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/eae2c7af3fbc4f4ab19f84bd1973b0cd 2023-03-24 21:56:21,761 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/b83e80911dce499084b719ec7c65399c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b83e80911dce499084b719ec7c65399c 2023-03-24 21:56:21,776 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/2965a8776c13468e85a3ddcc6e079f93 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/2965a8776c13468e85a3ddcc6e079f93 2023-03-24 21:56:21,783 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into b83e80911dce499084b719ec7c65399c(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:21,783 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,783 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 2965a8776c13468e85a3ddcc6e079f93(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:21,783 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694981705; duration=0sec 2023-03-24 21:56:21,783 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:21,783 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=12, startTime=1679694981705; duration=0sec 2023-03-24 21:56:21,783 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:21,783 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:21,783 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:21,784 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49324 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:21,785 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:21,785 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,785 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/aea33f7ccd504a3e8541d043e316d32c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=48.2 K 2023-03-24 21:56:21,785 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting aea33f7ccd504a3e8541d043e316d32c, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=567, earliestPutTs=1679694979826 2023-03-24 21:56:21,785 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5336c224592c4e7e894e390f192bddff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=603, earliestPutTs=1679694979864 2023-03-24 21:56:21,786 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting e781ba46430d4a54b4efbd19619dafcc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1679694980499 2023-03-24 21:56:21,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695041784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,790 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695041784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695041790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,795 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b62ac4c32e974fb799d36fae4a9df12c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981634 2023-03-24 21:56:21,805 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#573 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:21,822 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:21,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:21,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:21,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:21,825 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:21,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:21,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:21,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,826 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,827 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,828 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:21,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:21,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:21,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:21,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:21,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:21,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:21,835 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:21,835 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,836 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:21,836 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:21,836 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,836 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,837 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,838 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,838 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3386d41c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,839 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,842 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,842 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,842 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,843 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,843 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,846 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,846 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,847 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,850 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,860 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,861 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,861 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,861 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695041891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695041891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:21,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695041894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,961 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:21,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:21,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:21,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:21,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:21,965 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:21,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:21,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:21,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,965 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,967 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,967 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,967 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,967 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:21,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:21,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:21,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:21,968 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:21,969 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,969 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:21,969 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:21,969 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:21,969 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:21,971 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,971 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,972 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:21,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f6b6ecc[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:21,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:21,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,974 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,974 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:21,974 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:21,974 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:21,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,974 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:21,977 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,977 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:21,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,978 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:21,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:21,980 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:21,980 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:21,980 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:21,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,982 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:21,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:21,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,983 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:21,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,986 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,986 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,986 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:21,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:21,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,987 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:21,987 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:21,987 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:21,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:21,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:21,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,087 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:22,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:22,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:22,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:22,090 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:22,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:22,090 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:22,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,092 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:22,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,093 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:22,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:22,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:22,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:22,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:22,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:22,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:22,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:22,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:22,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:22,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,096 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695042093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:22,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695042094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:22,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d8f77f8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:22,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:22,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,099 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:22,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:22,100 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:22,100 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,100 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:22,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695042097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:22,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:22,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:22,104 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:22,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:22,104 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:22,105 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:22,105 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:22,106 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:22,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:22,115 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,115 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,116 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,116 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/01005c9263844a859fa08c49fb014264 2023-03-24 21:56:22,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=655 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/017eddc0739e4cf08b09b0ef73af75a1 2023-03-24 21:56:22,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/eae2c7af3fbc4f4ab19f84bd1973b0cd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd 2023-03-24 21:56:22,187 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd, entries=250, sequenceid=655, filesize=39.2 K 2023-03-24 21:56:22,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/01005c9263844a859fa08c49fb014264 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264 2023-03-24 21:56:22,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264, entries=150, sequenceid=655, filesize=11.9 K 2023-03-24 21:56:22,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/017eddc0739e4cf08b09b0ef73af75a1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1 2023-03-24 21:56:22,197 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1, entries=150, sequenceid=655, filesize=11.9 K 2023-03-24 21:56:22,197 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~220.88 KB/226176, currentSize=100.63 KB/103050 for 09571a8f4d95ff573f25f32562564d3a in 491ms, sequenceid=655, compaction requested=false 2023-03-24 21:56:22,197 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:22,216 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:22,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:22,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:22,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:22,219 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:22,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:22,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:22,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,221 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,221 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,222 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,222 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:22,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:22,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:22,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:22,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:22,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:22,223 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:22,223 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,223 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:22,224 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-24 21:56:22,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:22,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,224 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/ec1da27f10884f6c8ccaba286a4c93c7 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec1da27f10884f6c8ccaba286a4c93c7 2023-03-24 21:56:22,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:22,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,224 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:22,225 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,229 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into ec1da27f10884f6c8ccaba286a4c93c7(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:22,229 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:22,229 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694981706; duration=0sec 2023-03-24 21:56:22,229 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:22,244 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032403c71515945e4e8bb5514898373fbff9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032403c71515945e4e8bb5514898373fbff9 2023-03-24 21:56:22,257 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=675, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/d1d302c3c0b840528658107447ab050e 2023-03-24 21:56:22,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:22,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:22,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695042411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695042412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695042413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695042516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695042517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695042517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,674 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/9a7ad5e15a524270bf2bc2aca3ca6ac4 2023-03-24 21:56:22,682 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=675 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9a42ddf0fba2423ba4387611d05517a5 2023-03-24 21:56:22,686 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/d1d302c3c0b840528658107447ab050e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e 2023-03-24 21:56:22,690 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e, entries=150, sequenceid=675, filesize=25.4 K 2023-03-24 21:56:22,690 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/9a7ad5e15a524270bf2bc2aca3ca6ac4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4 2023-03-24 21:56:22,693 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4, entries=150, sequenceid=675, filesize=11.9 K 2023-03-24 21:56:22,694 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9a42ddf0fba2423ba4387611d05517a5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5 2023-03-24 21:56:22,697 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5, entries=150, sequenceid=675, filesize=11.9 K 2023-03-24 21:56:22,698 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 09571a8f4d95ff573f25f32562564d3a in 475ms, sequenceid=675, compaction requested=true 2023-03-24 21:56:22,698 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:22,698 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:22,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:22,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:22,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:22,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:22,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fa8b4a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:22,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:22,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,703 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:22,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:22,704 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:22,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:22,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:22,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,706 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:22,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:22,712 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:22,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:22,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:22,712 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:22,712 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:22,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,714 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,715 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,717 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,720 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:22,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,720 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,720 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:22,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,721 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:22,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,721 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:22,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-24 21:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,723 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:22,724 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:22,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695042745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695042747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695042747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,821 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:22,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:22,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:22,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:22,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:22,824 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:22,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:22,824 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:22,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,825 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,830 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,832 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:22,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:22,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:22,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:22,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:22,832 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:22,832 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:22,834 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,834 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:22,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:22,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5989dd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:22,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:22,836 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,837 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,837 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:22,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:22,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:22,837 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:22,838 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,838 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:22,839 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,839 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:22,841 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:22,841 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:22,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:22,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:22,841 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:22,841 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:22,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,844 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,844 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,846 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,848 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:22,849 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:22,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,848 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:22,849 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:22,849 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:22,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,849 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695042850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695042851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695042851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56574 deadline: 1679695042933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,940 DEBUG [Thread-3883] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:56:22,949 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:22,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:22,952 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:22,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:22,955 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:22,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:22,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:22,955 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,957 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:22,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:22,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:22,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:22,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:22,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:22,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:22,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:22,963 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:22,963 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,963 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:22,963 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:22,964 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:22,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:22,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:22,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:22,964 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:22,966 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,966 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,966 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:22,966 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.2.10:56586 deadline: 1679695042964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,966 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,968 DEBUG [Thread-3887] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., hostname=jenkins-hbase9.apache.org,45673,1679694819052, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor33.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-24 21:56:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14d09a14[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:22,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:22,970 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,973 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:22,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:22,973 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:22,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,973 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:22,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:22,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,975 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:22,975 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:22,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:22,977 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:22,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:22,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:22,977 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:22,978 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:22,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,980 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:22,981 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:22,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:22,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:22,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:22,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:22,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:22,984 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:22,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:22,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:22,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:22,984 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:22,984 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:22,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:22,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695043052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,056 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695043056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695043056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,085 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:23,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:23,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:23,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:23,088 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:23,088 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:23,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:23,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,089 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,091 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,091 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,092 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:23,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:23,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:23,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:23,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:23,093 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:23,093 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,093 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:23,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:23,094 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:23,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:23,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:23,094 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:23,095 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,095 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,096 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52a60fc1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:23,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:23,098 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,099 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:23,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:23,099 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:23,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,099 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:23,100 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:23,100 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,101 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:23,101 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:23,103 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:23,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:23,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:23,103 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:23,103 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:23,106 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,107 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,113 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:23,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:23,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,113 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,113 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:23,113 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:23,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:23,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,164 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324c4cb78b04626420088dbbc061bd939cc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c4cb78b04626420088dbbc061bd939cc 2023-03-24 21:56:23,213 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:23,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:23,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:23,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:23,217 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:23,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:23,217 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:23,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,218 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,219 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,219 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,220 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:23,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:23,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:23,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:23,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:23,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:23,221 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:23,221 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,221 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:23,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:23,222 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:23,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:23,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:23,222 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:23,227 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,227 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,228 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:23,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3576e04[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:23,229 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:23,229 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,230 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:23,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:23,231 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:23,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,231 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:23,233 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:23,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,233 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:23,236 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:23,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:23,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:23,236 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:23,236 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:23,238 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,238 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,238 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,238 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,239 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,248 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:23,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,249 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,249 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:23,249 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:23,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,349 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:23,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:23,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:23,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:23,352 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:23,352 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:23,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:23,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,353 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,355 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,355 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:23,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:23,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:23,356 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:23,356 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,356 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:23,357 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:23,357 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:23,357 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,357 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:23,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,357 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:23,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695043356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,358 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,358 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:23,358 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:23,358 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,358 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:23,358 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:23,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695043358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695043360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:23,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f8c08d2[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:23,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:23,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,363 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:23,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:23,363 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:23,363 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,364 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:23,366 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:23,366 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,366 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,366 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:23,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:23,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:23,368 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:23,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:23,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:23,369 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:23,369 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:23,370 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,370 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,370 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,371 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,372 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,374 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:23,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:23,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:23,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,375 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:23,375 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:23,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,475 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:23,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:23,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:23,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:23,479 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:23,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:23,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:23,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:23,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:23,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:23,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:23,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:23,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:23,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:23,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:23,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:23,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:23,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:23,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:23,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:23,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:23,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:23,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f9c3b0d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:23,489 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:23,489 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,490 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:23,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:23,490 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:23,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,490 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:23,491 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,491 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:23,491 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,491 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:23,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:23,494 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:23,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:23,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:23,494 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:23,494 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:23,495 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,495 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:23,495 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:23,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,503 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,515 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:23,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:23,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:23,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:23,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:23,519 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:23,519 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:23,519 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:23,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:23,570 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=691, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/86045fcd99f34148a02b230415da0807 2023-03-24 21:56:23,588 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/89672d132fd040df9b80929a7bae5062 2023-03-24 21:56:23,599 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=691 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f4d0cfcd6ca04d32986b3d2f1148cc09 2023-03-24 21:56:23,604 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/86045fcd99f34148a02b230415da0807 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807 2023-03-24 21:56:23,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807, entries=150, sequenceid=691, filesize=25.4 K 2023-03-24 21:56:23,608 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/89672d132fd040df9b80929a7bae5062 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062 2023-03-24 21:56:23,611 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062, entries=150, sequenceid=691, filesize=11.9 K 2023-03-24 21:56:23,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f4d0cfcd6ca04d32986b3d2f1148cc09 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09 2023-03-24 21:56:23,615 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09, entries=150, sequenceid=691, filesize=11.9 K 2023-03-24 21:56:23,616 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 09571a8f4d95ff573f25f32562564d3a in 893ms, sequenceid=691, compaction requested=true 2023-03-24 21:56:23,616 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:23,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:23,616 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:23,616 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:23,616 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:23,617 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:23,618 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118984 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:23,618 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:23,618 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,618 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/2965a8776c13468e85a3ddcc6e079f93, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=116.2 K 2023-03-24 21:56:23,618 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:23,618 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:23,618 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,618 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b83e80911dce499084b719ec7c65399c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=48.3 K 2023-03-24 21:56:23,619 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 2965a8776c13468e85a3ddcc6e079f93, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981634 2023-03-24 21:56:23,619 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:23,619 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting b83e80911dce499084b719ec7c65399c, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981634 2023-03-24 21:56:23,619 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 01005c9263844a859fa08c49fb014264, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1679694981679 2023-03-24 21:56:23,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:23,619 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting eae2c7af3fbc4f4ab19f84bd1973b0cd, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1679694981676 2023-03-24 21:56:23,620 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9a7ad5e15a524270bf2bc2aca3ca6ac4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694981726 2023-03-24 21:56:23,620 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d1d302c3c0b840528658107447ab050e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694981726 2023-03-24 21:56:23,620 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 89672d132fd040df9b80929a7bae5062, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:23,620 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 86045fcd99f34148a02b230415da0807, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:23,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:23,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:23,624 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:23,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:23,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:23,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,624 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,626 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:23,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:23,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,626 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:23,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:23,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:23,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:23,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:23,627 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:23,627 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:23,627 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:23,633 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#582 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:23,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:23,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:23,635 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:23,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:23,649 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#581 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:23,670 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/d7d4c1540a63420e82b39926149420ef as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/d7d4c1540a63420e82b39926149420ef 2023-03-24 21:56:23,678 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into d7d4c1540a63420e82b39926149420ef(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:23,678 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:23,678 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694983616; duration=0sec 2023-03-24 21:56:23,678 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:23,679 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:23,679 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:23,680 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:23,680 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:23,680 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec1da27f10884f6c8ccaba286a4c93c7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=48.3 K 2023-03-24 21:56:23,680 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ec1da27f10884f6c8ccaba286a4c93c7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=627, earliestPutTs=1679694981634 2023-03-24 21:56:23,681 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 017eddc0739e4cf08b09b0ef73af75a1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=655, earliestPutTs=1679694981679 2023-03-24 21:56:23,681 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9a42ddf0fba2423ba4387611d05517a5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=675, earliestPutTs=1679694981726 2023-03-24 21:56:23,690 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f4d0cfcd6ca04d32986b3d2f1148cc09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:23,693 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240e10ad17df1c4248ba7caded55e5faa4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240e10ad17df1c4248ba7caded55e5faa4 2023-03-24 21:56:23,701 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#584 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:23,743 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=708, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/8bcfd90b2619476baebe3c390fed966c 2023-03-24 21:56:23,754 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/9146d2ef58f449ca8bf0184e0dfa506d 2023-03-24 21:56:23,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:23,863 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:23,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695043876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,881 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 336 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695043877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695043878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695043979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,984 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695043982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:23,985 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:23,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 338 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695043982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,150 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/3e62d567a20a45b68bf874d66da90a99 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/3e62d567a20a45b68bf874d66da90a99 2023-03-24 21:56:24,153 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 3e62d567a20a45b68bf874d66da90a99(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:24,154 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:24,154 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694983617; duration=0sec 2023-03-24 21:56:24,154 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:24,163 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=708 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/a2c1aafcad9a46009ef98b3fb8a0a1eb 2023-03-24 21:56:24,167 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/8bcfd90b2619476baebe3c390fed966c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c 2023-03-24 21:56:24,170 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c, entries=150, sequenceid=708, filesize=25.4 K 2023-03-24 21:56:24,171 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/9146d2ef58f449ca8bf0184e0dfa506d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d 2023-03-24 21:56:24,174 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d, entries=150, sequenceid=708, filesize=11.9 K 2023-03-24 21:56:24,174 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/a2c1aafcad9a46009ef98b3fb8a0a1eb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb 2023-03-24 21:56:24,178 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb, entries=150, sequenceid=708, filesize=11.9 K 2023-03-24 21:56:24,178 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 09571a8f4d95ff573f25f32562564d3a in 543ms, sequenceid=708, compaction requested=false 2023-03-24 21:56:24,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:24,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,180 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,181 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3da6ef38[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,182 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,183 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:24,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:24,186 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:24,189 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,189 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,194 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,194 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,196 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,197 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,199 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,201 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,201 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,202 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,202 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,202 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695044204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695044208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695044208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,302 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,306 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,306 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,308 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:24,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,310 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695044309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,313 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,313 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,314 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695044313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695044314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b991c6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,316 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,317 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,317 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,317 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,317 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,319 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,319 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,322 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,322 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,322 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,324 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,325 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,326 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,330 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,329 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,330 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,330 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,330 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,330 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,331 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,430 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,434 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,435 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,436 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,436 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,437 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,438 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,439 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,439 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,442 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,442 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a5cd33b[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,445 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,445 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,447 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,447 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,450 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,450 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,450 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,451 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,452 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,453 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,454 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,456 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,456 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,457 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,457 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,457 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,499 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/e947ec4a388041b58948f3aabb0c4f7f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e947ec4a388041b58948f3aabb0c4f7f 2023-03-24 21:56:24,503 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into e947ec4a388041b58948f3aabb0c4f7f(size=26.3 K), total size for store is 51.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:24,503 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:24,503 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=12, startTime=1679694983616; duration=0sec 2023-03-24 21:56:24,504 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:24,513 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695044512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695044517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,519 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695044517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,557 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,559 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,560 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,562 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,562 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:24,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,564 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,564 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,564 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,564 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,564 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,567 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,567 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14a6d12e[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,569 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,570 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,570 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,571 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,571 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,574 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,574 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,581 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,583 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,583 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,584 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,584 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,584 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,584 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,586 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,586 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,602 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032474da80a1ba0f4f6c912696fb780094e8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032474da80a1ba0f4f6c912696fb780094e8 2023-03-24 21:56:24,684 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,686 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,687 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,687 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,688 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,689 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,689 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,690 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,691 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:24,691 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,691 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,691 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,691 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,692 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,693 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef323ad[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,698 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,699 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,699 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,700 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,702 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,702 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,704 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,709 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,709 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,809 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,813 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,815 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,816 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,816 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:24,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,817 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695044815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,819 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,819 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@578da614[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,821 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695044821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,822 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:24,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695044822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,822 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,823 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,823 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,824 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,824 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,827 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,827 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,827 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,827 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,830 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,831 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,832 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,835 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,835 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,835 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,835 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:24,835 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,836 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,936 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:24,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:24,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:24,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:24,939 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:24,939 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,939 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:24,939 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:24,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,940 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,941 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,941 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:24,942 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:24,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,942 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:24,942 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:24,942 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:24,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:24,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:24,942 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:24,943 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,943 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:24,943 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:24,943 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:24,943 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:24,944 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,944 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,945 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,945 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:24,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0472d5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:24,946 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:24,947 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,948 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:24,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:24,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,948 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:24,948 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:24,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:24,951 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,951 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:24,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:24,954 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:24,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:24,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:24,954 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:24,954 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:24,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:24,955 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:24,956 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:24,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,957 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:24,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:24,964 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:24,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:24,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:24,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:24,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:24,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:24,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:24,964 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:24,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:24,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:24,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:24,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:24,965 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:24,965 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:24,965 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,008 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=726, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/cb257d56a19d4fc0879a008bce5477cc 2023-03-24 21:56:25,020 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/aa8918ffc77b435dbae238c36f53bd8a 2023-03-24 21:56:25,030 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=726 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5a97903b047c4eccbe288294fa40cdd3 2023-03-24 21:56:25,034 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/cb257d56a19d4fc0879a008bce5477cc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc 2023-03-24 21:56:25,038 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc, entries=150, sequenceid=726, filesize=25.4 K 2023-03-24 21:56:25,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/aa8918ffc77b435dbae238c36f53bd8a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a 2023-03-24 21:56:25,041 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a, entries=150, sequenceid=726, filesize=11.9 K 2023-03-24 21:56:25,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5a97903b047c4eccbe288294fa40cdd3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3 2023-03-24 21:56:25,045 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3, entries=150, sequenceid=726, filesize=11.9 K 2023-03-24 21:56:25,046 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 09571a8f4d95ff573f25f32562564d3a in 860ms, sequenceid=726, compaction requested=true 2023-03-24 21:56:25,046 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:25,046 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:25,046 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:25,046 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:25,047 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78990 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:25,047 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:25,047 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,047 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e947ec4a388041b58948f3aabb0c4f7f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=77.1 K 2023-03-24 21:56:25,048 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:25,048 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:25,048 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,048 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/d7d4c1540a63420e82b39926149420ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.5 K 2023-03-24 21:56:25,048 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting e947ec4a388041b58948f3aabb0c4f7f, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:25,049 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 8bcfd90b2619476baebe3c390fed966c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1679694982731 2023-03-24 21:56:25,049 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting d7d4c1540a63420e82b39926149420ef, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:25,049 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting cb257d56a19d4fc0879a008bce5477cc, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:25,049 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9146d2ef58f449ca8bf0184e0dfa506d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1679694982731 2023-03-24 21:56:25,050 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting aa8918ffc77b435dbae238c36f53bd8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:25,058 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#591 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:25,061 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#590 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:25,065 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,072 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/0ec859d1b69f43ec94faa38c86eb0f2c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0ec859d1b69f43ec94faa38c86eb0f2c 2023-03-24 21:56:25,077 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into 0ec859d1b69f43ec94faa38c86eb0f2c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:25,077 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,077 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694985046; duration=0sec 2023-03-24 21:56:25,077 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:25,077 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:25,078 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:25,078 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:25,078 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,078 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/3e62d567a20a45b68bf874d66da90a99, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.5 K 2023-03-24 21:56:25,078 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3e62d567a20a45b68bf874d66da90a99, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=691, earliestPutTs=1679694982411 2023-03-24 21:56:25,079 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a2c1aafcad9a46009ef98b3fb8a0a1eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=708, earliestPutTs=1679694982731 2023-03-24 21:56:25,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,079 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5a97903b047c4eccbe288294fa40cdd3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:25,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,079 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,079 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,080 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,081 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,081 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,084 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,085 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#592 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:25,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,086 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,086 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:25,086 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,104 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/4abb533ad9a14d10b20e9d30e2b84c94 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4abb533ad9a14d10b20e9d30e2b84c94 2023-03-24 21:56:25,108 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 4abb533ad9a14d10b20e9d30e2b84c94(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:25,109 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,109 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694985046; duration=0sec 2023-03-24 21:56:25,109 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:25,124 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244188f0ff31eb44408663ac316f2df375 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244188f0ff31eb44408663ac316f2df375 2023-03-24 21:56:25,136 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=745, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/ca4f7cf9c9d647d9be2fdcedf2bb8794 2023-03-24 21:56:25,153 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/ce2c6a95bf3e4810951be693c182093d 2023-03-24 21:56:25,163 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=745 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/c6b00e8bf729442d98e8a26e63dfdbbb 2023-03-24 21:56:25,166 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/ca4f7cf9c9d647d9be2fdcedf2bb8794 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794 2023-03-24 21:56:25,170 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794, entries=150, sequenceid=745, filesize=25.4 K 2023-03-24 21:56:25,170 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/ce2c6a95bf3e4810951be693c182093d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d 2023-03-24 21:56:25,174 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d, entries=150, sequenceid=745, filesize=11.9 K 2023-03-24 21:56:25,174 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/c6b00e8bf729442d98e8a26e63dfdbbb as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb 2023-03-24 21:56:25,177 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb, entries=150, sequenceid=745, filesize=11.9 K 2023-03-24 21:56:25,178 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 91ms, sequenceid=745, compaction requested=false 2023-03-24 21:56:25,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,178 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:25,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:25,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:25,178 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:25,179 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,179 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,180 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:25,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3909ef3a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:25,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:25,181 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,184 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:25,184 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:25,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,184 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:25,185 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:25,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,185 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:25,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:25,188 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:25,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:25,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:25,188 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:25,188 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:25,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,190 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,190 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,192 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,194 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:25,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:25,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:25,194 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,195 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:25,195 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,295 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,298 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,298 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,299 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,302 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,302 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,303 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,303 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,304 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,304 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,304 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,304 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:25,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:25,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:25,305 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:25,306 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,306 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,307 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:25,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:25,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ee05dfd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:25,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:25,308 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,309 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:25,309 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:25,310 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:25,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,310 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:25,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:25,311 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,311 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:25,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:25,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:25,314 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:25,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:25,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:25,314 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:25,314 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:25,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,316 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,317 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,318 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,320 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:25,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:25,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:25,320 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,321 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:25,321 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,321 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:25,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:25,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:25,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:25,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:25,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:25,365 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303244b37da92057d405ba8e86ea4ad7d1d97 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b37da92057d405ba8e86ea4ad7d1d97 2023-03-24 21:56:25,370 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=758, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/9bd29b07618a43fc822db6413be53cd5 2023-03-24 21:56:25,378 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/7cba042e6dbc41c99d92182048bb2b18 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694985369, max timestamp=1679694985378Num uniques 859; 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694985369, max timestamp=1679694985378Num uniques 859; 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679694985369, max timestamp=1679694985378Num uniques 859; 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:25,380 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-24 21:56:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695045402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695045403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695045403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,421 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,424 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,424 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,425 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,426 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,426 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,427 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,428 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,428 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,428 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:25,428 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:25,428 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:25,430 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,430 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,431 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:25,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7f133[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:25,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:25,432 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,434 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:25,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:25,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,434 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:25,434 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:25,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:25,435 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,435 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,436 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:25,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:25,438 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:25,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:25,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:25,438 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:25,438 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,440 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,441 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,442 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,444 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:25,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:25,444 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,445 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:25,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,445 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:25,445 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,445 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695045508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695045508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695045509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,545 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,549 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,549 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,555 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,556 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,556 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,556 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,556 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,556 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,557 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,557 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:25,558 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:25,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:25,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:25,558 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:25,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,559 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,560 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,561 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:25,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ccdb9a0[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:25,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:25,561 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,563 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:25,563 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:25,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,563 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:25,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:25,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,570 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:25,571 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:25,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:25,573 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:25,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:25,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:25,573 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:25,573 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:25,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,575 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,575 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,577 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,580 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,579 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:25,580 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,580 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:25,580 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,580 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,680 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,683 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,684 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,685 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,685 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,686 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,686 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,687 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,687 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,687 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,687 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,687 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,688 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:25,688 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:25,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:25,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:25,693 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a4be13c[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:25,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:25,693 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,694 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:25,694 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:25,695 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:25,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,695 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:25,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:25,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:25,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:25,699 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:25,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:25,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:25,699 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:25,699 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:25,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:25,700 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:25,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,701 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:25,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:25,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:25,706 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:25,705 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:25,706 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,706 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:25,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:25,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:25,706 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:25,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695045710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,713 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695045712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:25,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695045712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=758 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f0da2a7b57874bae92608dab44a7514f 2023-03-24 21:56:25,796 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/9bd29b07618a43fc822db6413be53cd5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5 2023-03-24 21:56:25,800 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5, entries=350, sequenceid=758, filesize=52.9 K 2023-03-24 21:56:25,800 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/7cba042e6dbc41c99d92182048bb2b18 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18 2023-03-24 21:56:25,804 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18, entries=150, sequenceid=758, filesize=11.9 K 2023-03-24 21:56:25,804 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f0da2a7b57874bae92608dab44a7514f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f 2023-03-24 21:56:25,806 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:25,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:25,808 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f, entries=150, sequenceid=758, filesize=11.9 K 2023-03-24 21:56:25,808 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 09571a8f4d95ff573f25f32562564d3a in 455ms, sequenceid=758, compaction requested=true 2023-03-24 21:56:25,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:25,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:25,809 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:56:25,809 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-03-24 21:56:25,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:25,809 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-24 21:56:25,809 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-24 21:56:25,809 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. because compaction request was cancelled 2023-03-24 21:56:25,810 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:25,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:25,810 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:25,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:25,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:25,810 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:25,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,810 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:25,810 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,811 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,811 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0ec859d1b69f43ec94faa38c86eb0f2c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.6 K 2023-03-24 21:56:25,811 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0ec859d1b69f43ec94faa38c86eb0f2c, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:25,811 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ce2c6a95bf3e4810951be693c182093d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1679694984201 2023-03-24 21:56:25,812 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:25,812 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 7cba042e6dbc41c99d92182048bb2b18, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1679694985349 2023-03-24 21:56:25,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:25,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:25,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:25,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:25,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:25,813 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:25,813 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:25,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:25,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:25,813 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:25,815 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:25,815 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:25,815 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:25,815 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:25,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:25,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:25,817 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#599 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:25,888 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/c0758f4d2fd14cacb7ecee4dace2b081 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c0758f4d2fd14cacb7ecee4dace2b081 2023-03-24 21:56:25,892 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into c0758f4d2fd14cacb7ecee4dace2b081(size=26.4 K), total size for store is 104.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:25,892 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,892 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694985046; duration=0sec 2023-03-24 21:56:25,892 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:25,892 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:25,893 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:25,893 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:25,894 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:25,894 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4abb533ad9a14d10b20e9d30e2b84c94, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=36.6 K 2023-03-24 21:56:25,894 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4abb533ad9a14d10b20e9d30e2b84c94, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:25,894 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c6b00e8bf729442d98e8a26e63dfdbbb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1679694984201 2023-03-24 21:56:25,895 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f0da2a7b57874bae92608dab44a7514f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1679694985349 2023-03-24 21:56:25,901 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#601 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:25,912 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/7d61d4c179dd4212992ee46a707d7a79 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/7d61d4c179dd4212992ee46a707d7a79 2023-03-24 21:56:25,916 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 7d61d4c179dd4212992ee46a707d7a79(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:25,916 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:25,916 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694985809; duration=0sec 2023-03-24 21:56:25,916 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:26,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:26,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:26,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695046023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695046024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695046024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695046126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695046127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,130 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695046128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,240 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/a18eaee9db224918a3e9f4e4a0f358bf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a18eaee9db224918a3e9f4e4a0f358bf 2023-03-24 21:56:26,244 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into a18eaee9db224918a3e9f4e4a0f358bf(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:26,244 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:26,244 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694985809; duration=0sec 2023-03-24 21:56:26,244 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:26,247 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324b158ae4699ba4ab593d2b91bac111708 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b158ae4699ba4ab593d2b91bac111708 2023-03-24 21:56:26,253 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=795, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/85eedf0a20bc41e59d8b09581b84c29c 2023-03-24 21:56:26,261 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8162645e573640cdb91ad1f66c4d5765 2023-03-24 21:56:26,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695046330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695046330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695046331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 380 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695046634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695046635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:26,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695046635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,671 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=795 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/af66deb7cf204b3b86dc0d5fbfbd2a44 2023-03-24 21:56:26,675 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/85eedf0a20bc41e59d8b09581b84c29c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c 2023-03-24 21:56:26,678 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c, entries=150, sequenceid=795, filesize=25.4 K 2023-03-24 21:56:26,679 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8162645e573640cdb91ad1f66c4d5765 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765 2023-03-24 21:56:26,683 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765, entries=150, sequenceid=795, filesize=11.9 K 2023-03-24 21:56:26,684 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/af66deb7cf204b3b86dc0d5fbfbd2a44 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44 2023-03-24 21:56:26,687 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44, entries=150, sequenceid=795, filesize=11.9 K 2023-03-24 21:56:26,687 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 09571a8f4d95ff573f25f32562564d3a in 872ms, sequenceid=795, compaction requested=true 2023-03-24 21:56:26,687 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:26,688 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:26,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:26,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:26,688 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:26,691 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:26,692 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,692 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,693 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:26,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76f8523d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:26,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:26,694 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:26,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,696 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:26,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:26,696 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:26,698 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:26,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,698 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:26,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:26,702 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:26,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:26,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:26,702 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:26,702 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:26,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,704 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:26,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,705 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,706 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:26,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:26,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:26,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,708 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:26,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:26,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:26,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:26,708 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,709 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:26,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:26,709 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,709 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:26,809 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:26,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:26,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:26,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:26,812 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:26,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:26,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:26,812 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:26,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,813 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,814 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:26,814 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,815 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:26,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,815 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:26,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:26,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:26,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:26,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:26,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:26,816 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:26,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:26,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:26,816 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:26,816 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:26,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:26,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:26,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:26,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:26,817 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:26,831 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324a2b30de148a94d809d342f2f3410d165 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a2b30de148a94d809d342f2f3410d165 2023-03-24 21:56:26,836 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=808, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/ab5aa0632828451fb3e951db7b636daf 2023-03-24 21:56:26,847 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/f75f21f3cb8c41a3b7ae42852962020b 2023-03-24 21:56:26,855 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=808 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fb3465ea7a554d1890da13e011463e61 2023-03-24 21:56:26,858 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/ab5aa0632828451fb3e951db7b636daf as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf 2023-03-24 21:56:26,861 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf, entries=100, sequenceid=808, filesize=18.6 K 2023-03-24 21:56:26,862 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/f75f21f3cb8c41a3b7ae42852962020b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b 2023-03-24 21:56:26,865 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b, entries=100, sequenceid=808, filesize=9.5 K 2023-03-24 21:56:26,866 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/fb3465ea7a554d1890da13e011463e61 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61 2023-03-24 21:56:26,870 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61, entries=100, sequenceid=808, filesize=9.5 K 2023-03-24 21:56:26,870 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 54ms, sequenceid=808, compaction requested=true 2023-03-24 21:56:26,870 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:26,870 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:26,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:26,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:26,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:26,871 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:26,876 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,876 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,876 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,876 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:26,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:26,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dad4bb8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:26,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:26,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,879 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:26,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:26,879 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:26,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,879 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:26,880 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,880 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:26,881 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,881 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:26,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:26,883 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:26,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:26,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:26,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:26,884 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:26,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:26,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:26,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:26,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:26,890 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,891 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:26,894 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:26,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:26,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:26,894 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,894 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:26,894 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:26,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:26,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:26,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:26,994 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:26,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:26,996 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:26,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:26,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:26,998 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:26,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:26,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:26,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:26,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:26,998 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:26,999 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,001 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,001 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,001 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,001 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:27,002 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:27,002 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:27,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,004 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:27,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:27,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7ceb14[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:27,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:27,006 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,008 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:27,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:27,009 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:27,010 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,010 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:27,010 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,011 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:27,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:27,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:27,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:27,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:27,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:27,011 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:27,011 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:27,012 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,012 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,013 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,013 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,014 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,016 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,017 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:27,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,017 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,017 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:27,017 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:27,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:27,117 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:27,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:27,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:27,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:27,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:27,121 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:27,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:27,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:27,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,121 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,124 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,125 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,125 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,125 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,126 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,126 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,126 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:27,126 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:27,126 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:27,127 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:27,128 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,128 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,128 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,128 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:27,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92baba6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:27,130 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:27,131 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,132 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,132 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:27,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:27,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:27,132 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:27,133 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,133 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:27,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,134 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:27,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:27,136 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:27,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:27,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:27,136 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:27,136 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:27,140 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,140 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,140 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,141 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,142 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,144 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,144 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:27,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:27,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:27,144 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,144 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,145 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,145 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,145 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:27,174 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:27,174 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:27,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:27,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:27,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:27,175 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:27,175 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:27,186 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303240bc0d229730440e5a9e6d3f9b46eea72 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc0d229730440e5a9e6d3f9b46eea72 2023-03-24 21:56:27,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694987187, max timestamp=1679694987199Num uniques 859; 2023-03-24 21:56:27,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694987187, max timestamp=1679694987199Num uniques 859; 2023-03-24 21:56:27,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:27,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694987187, max timestamp=1679694987199Num uniques 859; 2023-03-24 21:56:27,201 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:27,202 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:27,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 396 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695047222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,227 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694987215, max timestamp=1679694987225Num uniques 1018; 2023-03-24 21:56:27,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:27,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694987215, max timestamp=1679694987225Num uniques 1018; 2023-03-24 21:56:27,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:27,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679694987215, max timestamp=1679694987225Num uniques 1018; 2023-03-24 21:56:27,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-24 21:56:27,245 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:27,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:27,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:27,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:27,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:27,249 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:27,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:27,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:27,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,250 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,251 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,252 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,252 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,252 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,252 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:27,253 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:27,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:27,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:27,253 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:27,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,255 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:27,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d767904[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:27,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:27,257 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,259 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:27,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:27,259 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:27,259 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:27,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679694987253, max timestamp=1679694987262Num uniques 1027; 2023-03-24 21:56:27,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679694987253, max timestamp=1679694987262Num uniques 1027; 2023-03-24 21:56:27,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:27,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:27,265 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679694987253, max timestamp=1679694987262Num uniques 1027; 2023-03-24 21:56:27,266 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-8] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-03-24 21:56:27,266 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:27,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,266 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:27,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:27,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:27,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:27,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:27,269 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:27,269 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:27,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,270 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,271 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,273 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:27,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:27,275 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,275 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,276 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:27,276 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:27,276 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:27,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695047294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695047295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695047329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,376 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:27,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:27,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:27,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:27,380 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:27,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:27,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:27,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,380 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,383 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,385 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,385 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,385 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:27,385 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:27,385 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:27,387 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,388 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:27,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a210599[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:27,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:27,389 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,391 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:27,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:27,391 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:27,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,391 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:27,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:27,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,392 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:27,393 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:27,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:27,395 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:27,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:27,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:27,395 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:27,395 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:27,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695047398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,398 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,399 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,401 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,404 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:27,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:27,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,404 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,404 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,404 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:27,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695047400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,404 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:27,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,407 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:27,504 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:27,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:27,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:27,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:27,507 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:27,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:27,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:27,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,508 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,509 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,510 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,511 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,511 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,511 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:27,511 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:27,511 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:27,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:27,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:27,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61fe7982[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:27,514 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:27,514 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,516 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:27,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:27,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,517 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:27,517 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:27,518 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:27,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,518 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:27,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:27,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:27,521 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:27,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:27,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:27,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:27,521 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:27,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,522 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:27,523 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:27,523 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,524 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:27,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:27,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,528 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:27,528 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:27,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:27,527 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:27,527 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:27,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:27,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:27,528 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695047533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,591 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=820, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/797ccfd6eecc4bd085dcd6a5073c9ec8 2023-03-24 21:56:27,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5d1dd3cd74d54c61b913ae178f0d1a0c 2023-03-24 21:56:27,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 308 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695047601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=820 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f0bb884e7b6f47e2a6ef08d365128341 2023-03-24 21:56:27,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695047608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,613 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/797ccfd6eecc4bd085dcd6a5073c9ec8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8 2023-03-24 21:56:27,617 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8, entries=200, sequenceid=820, filesize=32.3 K 2023-03-24 21:56:27,617 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5d1dd3cd74d54c61b913ae178f0d1a0c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c 2023-03-24 21:56:27,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c, entries=150, sequenceid=820, filesize=11.9 K 2023-03-24 21:56:27,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/f0bb884e7b6f47e2a6ef08d365128341 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341 2023-03-24 21:56:27,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341, entries=150, sequenceid=820, filesize=11.9 K 2023-03-24 21:56:27,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 09571a8f4d95ff573f25f32562564d3a in 452ms, sequenceid=820, compaction requested=true 2023-03-24 21:56:27,626 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:27,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:27,626 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-24 21:56:27,626 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:27,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:27,626 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:27,628 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:27,628 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:27,628 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:27,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:27,628 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,628 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a18eaee9db224918a3e9f4e4a0f358bf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.2 K 2023-03-24 21:56:27,629 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 185390 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-24 21:56:27,629 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting a18eaee9db224918a3e9f4e4a0f358bf, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1679694985349 2023-03-24 21:56:27,629 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:27,629 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,629 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c0758f4d2fd14cacb7ecee4dace2b081, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=181.0 K 2023-03-24 21:56:27,629 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8162645e573640cdb91ad1f66c4d5765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679694985397 2023-03-24 21:56:27,629 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting c0758f4d2fd14cacb7ecee4dace2b081, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=726, earliestPutTs=1679694983876 2023-03-24 21:56:27,630 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting f75f21f3cb8c41a3b7ae42852962020b, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694986014 2023-03-24 21:56:27,630 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ca4f7cf9c9d647d9be2fdcedf2bb8794, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=745, earliestPutTs=1679694984201 2023-03-24 21:56:27,630 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5d1dd3cd74d54c61b913ae178f0d1a0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:27,630 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 9bd29b07618a43fc822db6413be53cd5, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1679694985335 2023-03-24 21:56:27,631 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 85eedf0a20bc41e59d8b09581b84c29c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679694985397 2023-03-24 21:56:27,631 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting ab5aa0632828451fb3e951db7b636daf, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694986014 2023-03-24 21:56:27,632 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 797ccfd6eecc4bd085dcd6a5073c9ec8, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:27,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:27,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:27,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:27,635 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:27,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:27,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:27,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,635 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,637 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:27,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:27,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:27,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,637 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:27,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:27,639 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:27,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:27,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:27,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:27,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:27,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:27,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:27,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:27,640 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:27,640 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=134.18 KB heapSize=378.61 KB 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:27,640 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:27,641 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:27,641 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:27,664 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#611 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:27,670 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303249e9ddbe6a1264f07adff5b810db9240c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e9ddbe6a1264f07adff5b810db9240c 2023-03-24 21:56:27,688 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/a5b693db2ffd42a7ac687e0317350fd3 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a5b693db2ffd42a7ac687e0317350fd3 2023-03-24 21:56:27,692 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into a5b693db2ffd42a7ac687e0317350fd3(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:27,692 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:27,692 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=10, startTime=1679694987626; duration=0sec 2023-03-24 21:56:27,692 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:27,692 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:27,696 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47356 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:27,696 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:27,696 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:27,696 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/7d61d4c179dd4212992ee46a707d7a79, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.2 K 2023-03-24 21:56:27,696 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 7d61d4c179dd4212992ee46a707d7a79, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=758, earliestPutTs=1679694985349 2023-03-24 21:56:27,697 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting af66deb7cf204b3b86dc0d5fbfbd2a44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=795, earliestPutTs=1679694985397 2023-03-24 21:56:27,697 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting fb3465ea7a554d1890da13e011463e61, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=808, earliestPutTs=1679694986014 2023-03-24 21:56:27,697 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting f0bb884e7b6f47e2a6ef08d365128341, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:27,704 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#613 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:27,712 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/4ce7dd57ffe14109b609c961ac2755b0 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4ce7dd57ffe14109b609c961ac2755b0 2023-03-24 21:56:27,716 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 4ce7dd57ffe14109b609c961ac2755b0(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:27,716 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:27,716 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694987626; duration=0sec 2023-03-24 21:56:27,716 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:27,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:27,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:27,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 409 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695047865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 310 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695047906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695047912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:27,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:27,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 411 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695047968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,050 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/9b89a6425c0f41ac969796ad8b44a5fd as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9b89a6425c0f41ac969796ad8b44a5fd 2023-03-24 21:56:28,054 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into 9b89a6425c0f41ac969796ad8b44a5fd(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:28,054 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:28,054 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694987626; duration=0sec 2023-03-24 21:56:28,054 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:28,077 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=890, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/a2e8217f7b6740e1a1b75ed2086eaf9a 2023-03-24 21:56:28,095 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=890 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5ba4e331df0c4504b7d880bf559d3fb5 2023-03-24 21:56:28,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:28,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 413 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695048171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:28,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695048414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:28,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695048417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,473 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:28,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 415 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695048472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,517 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=890 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/0241311d4d3e4c7c8019bb33aeb190e5 2023-03-24 21:56:28,521 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/a2e8217f7b6740e1a1b75ed2086eaf9a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a 2023-03-24 21:56:28,525 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a, entries=150, sequenceid=890, filesize=25.4 K 2023-03-24 21:56:28,525 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/5ba4e331df0c4504b7d880bf559d3fb5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5 2023-03-24 21:56:28,529 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5, entries=150, sequenceid=890, filesize=11.9 K 2023-03-24 21:56:28,529 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/0241311d4d3e4c7c8019bb33aeb190e5 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5 2023-03-24 21:56:28,532 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5, entries=150, sequenceid=890, filesize=11.9 K 2023-03-24 21:56:28,533 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~378.56 KB/387648, currentSize=46.96 KB/48090 for 09571a8f4d95ff573f25f32562564d3a in 893ms, sequenceid=890, compaction requested=false 2023-03-24 21:56:28,533 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:28,533 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:28,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:28,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:28,533 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:28,535 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,535 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,536 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:28,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35dc9880[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:28,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:28,537 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,538 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,538 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:28,538 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:28,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,538 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:28,538 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:28,539 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:28,539 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,539 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,540 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:28,542 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:28,542 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:28,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:28,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:28,542 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:28,542 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:28,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,551 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,551 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,552 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:28,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:28,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,558 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:28,558 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:28,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,558 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:28,558 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,559 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,659 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:28,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:28,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:28,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:28,662 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:28,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:28,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:28,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,662 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,664 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,664 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:28,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:28,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:28,665 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:28,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:28,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,665 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:28,666 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:28,666 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:28,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:28,666 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:28,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:28,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:28,666 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:28,667 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:28,667 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:28,667 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:28,677 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e08032116c974601844772ffd57366c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e08032116c974601844772ffd57366c4 2023-03-24 21:56:28,683 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=903, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/d391fbf85ecd4d79b86fbcc11ae7e61e 2023-03-24 21:56:28,697 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=903 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/37085b91138d4cb7945e91d5d1c62be4 2023-03-24 21:56:28,713 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=903 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/32e2159d51df43ecb8fdbe90c1f11246 2023-03-24 21:56:28,717 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/d391fbf85ecd4d79b86fbcc11ae7e61e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e 2023-03-24 21:56:28,722 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e, entries=100, sequenceid=903, filesize=18.6 K 2023-03-24 21:56:28,723 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/37085b91138d4cb7945e91d5d1c62be4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4 2023-03-24 21:56:28,726 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4, entries=100, sequenceid=903, filesize=9.5 K 2023-03-24 21:56:28,726 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/32e2159d51df43ecb8fdbe90c1f11246 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246 2023-03-24 21:56:28,733 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246, entries=100, sequenceid=903, filesize=9.5 K 2023-03-24 21:56:28,733 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 67ms, sequenceid=903, compaction requested=true 2023-03-24 21:56:28,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:28,733 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:28,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:28,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:28,734 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:28,735 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,735 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,736 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:28,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bec7966[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:28,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:28,737 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,740 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:28,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:28,741 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:28,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:28,742 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:28,742 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,742 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,742 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:28,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:28,745 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:28,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:28,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:28,745 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:28,745 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:28,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,747 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,748 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,749 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,751 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:28,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:28,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,752 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:28,752 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:28,853 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:28,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:28,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:28,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:28,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:28,856 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:28,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:28,856 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:28,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,857 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,859 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,859 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,860 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:28,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:28,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:28,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:28,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:28,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:28,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:28,861 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:28,861 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,861 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:28,861 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:28,861 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:28,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:28,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:28,862 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:28,863 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,863 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,864 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:28,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2973ae4f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:28,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:28,865 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,866 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:28,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:28,866 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:28,866 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,866 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:28,868 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:28,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,868 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:28,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:28,871 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:28,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:28,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:28,871 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:28,871 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:28,873 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,874 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,874 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,874 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,875 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,877 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,877 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:28,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:28,878 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:28,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,878 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:28,878 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,878 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,978 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:28,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:28,980 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:28,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:28,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:28,982 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,982 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,983 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:28,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:28,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,984 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:28,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:28,985 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:28,985 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:28,985 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:28,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:28,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:28,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:28,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:28,985 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:28,985 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,985 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:28,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:28,986 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:28,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:28,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:28,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:28,986 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:28,987 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,987 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,988 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:28,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bd3f08a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:28,989 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:28,990 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,992 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:28,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,992 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:28,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:28,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:28,992 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:28,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:28,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:28,994 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,994 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:28,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:28,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:28,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:28,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:28,996 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:28,996 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:28,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:28,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:28,998 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:28,999 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:28,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:28,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,000 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,003 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,003 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,004 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,004 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:29,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-24 21:56:29,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:29,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:29,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:29,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:29,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:29,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694989050, max timestamp=1679694989081Num uniques 859; 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694989050, max timestamp=1679694989081Num uniques 859; 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694989050, max timestamp=1679694989081Num uniques 859; 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:29,083 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:29,104 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,107 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,107 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,108 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,108 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,109 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,110 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,110 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,111 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,112 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,112 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,113 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a1a67e8[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,114 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,116 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,116 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,116 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,117 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,119 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,120 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,122 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,122 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,123 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,124 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,126 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,127 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,129 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,129 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,129 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,130 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,130 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 458 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695049141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,230 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,233 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,233 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,234 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,235 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,235 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,236 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,236 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,236 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,237 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,237 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,239 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,239 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,240 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e50484[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,241 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,242 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,242 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,243 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,243 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,244 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,244 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,246 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,247 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,247 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,247 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 460 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695049246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,248 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,249 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,250 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,254 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,254 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,254 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,254 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,255 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,255 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,255 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,355 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,359 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,361 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,361 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,362 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,371 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,378 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,378 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,378 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,378 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,378 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,379 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,380 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,380 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,381 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@650c541[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,383 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,384 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,384 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,384 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,387 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,387 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,390 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,390 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,390 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,392 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,392 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,392 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,393 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,394 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,400 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,400 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,400 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,400 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,419 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695049417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695049430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,441 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303247b93193213b043309a4b2f83357544bf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b93193213b043309a4b2f83357544bf 2023-03-24 21:56:29,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 462 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695049449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,500 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,504 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,504 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,506 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,506 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,507 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,507 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,507 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,507 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,508 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,508 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,509 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,509 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,510 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,512 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@222332a1[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,512 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,512 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,513 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,513 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,513 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,513 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,515 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,515 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,518 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,518 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,518 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,518 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,520 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,521 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,522 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,524 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,525 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,525 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,526 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,526 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,626 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,630 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,631 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,631 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,632 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,635 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,635 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,636 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,638 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,638 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,639 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bbf342a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,640 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,640 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,641 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,642 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,642 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,643 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,643 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,643 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,643 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,646 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,646 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,648 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,648 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,650 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,653 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,654 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,653 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,654 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,654 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:29,754 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:29,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 464 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695049754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,757 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,757 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,759 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,760 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,760 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,760 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,761 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,761 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,761 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:29,761 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:29,761 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:29,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4c2fd[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:29,764 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:29,765 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,766 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:29,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:29,766 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:29,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,766 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:29,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:29,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,769 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,769 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:29,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:29,771 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:29,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:29,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:29,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:29,772 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:29,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:29,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,775 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,776 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,778 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:29,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:29,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:29,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:29,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:29,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,778 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:29,779 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,778 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:29,779 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:29,846 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=915, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/500809aaaf8e471c8bb3fda25a65597f 2023-03-24 21:56:29,856 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=915 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/cd5182a7ca31437ca702685e1ed2959c 2023-03-24 21:56:29,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=915 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/93db49fa9f9c4626abe96ac082566886 2023-03-24 21:56:29,869 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/500809aaaf8e471c8bb3fda25a65597f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f 2023-03-24 21:56:29,872 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f, entries=200, sequenceid=915, filesize=32.3 K 2023-03-24 21:56:29,873 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/cd5182a7ca31437ca702685e1ed2959c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c 2023-03-24 21:56:29,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c, entries=150, sequenceid=915, filesize=11.9 K 2023-03-24 21:56:29,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/93db49fa9f9c4626abe96ac082566886 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886 2023-03-24 21:56:29,879 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:29,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:29,881 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886, entries=150, sequenceid=915, filesize=11.9 K 2023-03-24 21:56:29,882 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 09571a8f4d95ff573f25f32562564d3a in 853ms, sequenceid=915, compaction requested=true 2023-03-24 21:56:29,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:29,882 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:29,882 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-24 21:56:29,882 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:29,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:29,883 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:29,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:29,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:29,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,884 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,884 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105334 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:29,884 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:29,884 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,884 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a5b693db2ffd42a7ac687e0317350fd3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=102.9 K 2023-03-24 21:56:29,884 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting a5b693db2ffd42a7ac687e0317350fd3, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:29,884 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:29,885 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:29,885 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,885 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9b89a6425c0f41ac969796ad8b44a5fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.4 K 2023-03-24 21:56:29,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,885 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting a2e8217f7b6740e1a1b75ed2086eaf9a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1679694987282 2023-03-24 21:56:29,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:29,885 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:29,885 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:29,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:29,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:29,885 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9b89a6425c0f41ac969796ad8b44a5fd, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:29,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting d391fbf85ecd4d79b86fbcc11ae7e61e, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1679694987861 2023-03-24 21:56:29,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,886 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 5ba4e331df0c4504b7d880bf559d3fb5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1679694987282 2023-03-24 21:56:29,886 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 500809aaaf8e471c8bb3fda25a65597f, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694988988 2023-03-24 21:56:29,886 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:29,886 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 37085b91138d4cb7945e91d5d1c62be4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1679694987861 2023-03-24 21:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:29,887 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:29,887 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting cd5182a7ca31437ca702685e1ed2959c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694989019 2023-03-24 21:56:29,887 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:29,887 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:29,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:29,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:29,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:29,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:29,887 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:29,888 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:29,888 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:29,896 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#624 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:29,908 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#623 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:29,924 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/ea288690a1674ac4adb0a83bb7dfffa1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ea288690a1674ac4adb0a83bb7dfffa1 2023-03-24 21:56:29,928 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into ea288690a1674ac4adb0a83bb7dfffa1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:29,928 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:29,928 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=12, startTime=1679694989882; duration=0sec 2023-03-24 21:56:29,928 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:29,928 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-24 21:56:29,929 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47492 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-24 21:56:29,929 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:29,929 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:29,929 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4ce7dd57ffe14109b609c961ac2755b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=46.4 K 2023-03-24 21:56:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4ce7dd57ffe14109b609c961ac2755b0, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=820, earliestPutTs=1679694987154 2023-03-24 21:56:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 0241311d4d3e4c7c8019bb33aeb190e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=890, earliestPutTs=1679694987282 2023-03-24 21:56:29,930 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 32e2159d51df43ecb8fdbe90c1f11246, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=903, earliestPutTs=1679694987861 2023-03-24 21:56:29,931 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 93db49fa9f9c4626abe96ac082566886, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694989019 2023-03-24 21:56:29,938 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#625 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:29,940 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/da6305bbcac14ba7912447f9f49eb58e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/da6305bbcac14ba7912447f9f49eb58e 2023-03-24 21:56:29,944 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into da6305bbcac14ba7912447f9f49eb58e(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:29,944 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:29,944 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=12, startTime=1679694989882; duration=0sec 2023-03-24 21:56:29,944 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:30,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:30,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:30,283 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:30,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 473 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695050281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,307 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303242ca67c0cf3ea4a62a1f47be74cbec6d6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242ca67c0cf3ea4a62a1f47be74cbec6d6 2023-03-24 21:56:30,312 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=952, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/4cc74bac77ed4b8186e02ebeb110274a 2023-03-24 21:56:30,324 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/686fa8bebc3a44798379f9c3f72e0e7b 2023-03-24 21:56:30,333 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=952 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/ec0792771fae4f4c917377b88ae7543a 2023-03-24 21:56:30,336 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/4cc74bac77ed4b8186e02ebeb110274a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a 2023-03-24 21:56:30,340 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a, entries=150, sequenceid=952, filesize=25.4 K 2023-03-24 21:56:30,341 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/686fa8bebc3a44798379f9c3f72e0e7b as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b 2023-03-24 21:56:30,344 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b, entries=150, sequenceid=952, filesize=11.9 K 2023-03-24 21:56:30,345 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/ec0792771fae4f4c917377b88ae7543a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a 2023-03-24 21:56:30,349 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a, entries=150, sequenceid=952, filesize=11.9 K 2023-03-24 21:56:30,350 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 09571a8f4d95ff573f25f32562564d3a in 462ms, sequenceid=952, compaction requested=false 2023-03-24 21:56:30,350 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:30,350 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:30,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:30,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:30,350 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:30,352 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/4f7f1229d567447e8bf919ff5151aede as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4f7f1229d567447e8bf919ff5151aede 2023-03-24 21:56:30,356 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 4f7f1229d567447e8bf919ff5151aede(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:30,356 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:30,356 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=12, startTime=1679694989882; duration=0sec 2023-03-24 21:56:30,356 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,356 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:30,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,357 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,357 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,357 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:30,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11e754d[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:30,358 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:30,359 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,360 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:30,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:30,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:30,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,360 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:30,361 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:30,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:30,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:30,364 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:30,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:30,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:30,365 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:30,365 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:30,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,367 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,370 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,374 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,375 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:30,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:30,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,376 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:30,376 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:30,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:30,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-24 21:56:30,385 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:30,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:30,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:30,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:30,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:30,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:30,399 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324235500096eb442a0a7e21f0fc9f9f2aa to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324235500096eb442a0a7e21f0fc9f9f2aa 2023-03-24 21:56:30,404 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=966, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/96d64de0904e4457817df6db3e38243a 2023-03-24 21:56:30,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694990420, max timestamp=1679694990437Num uniques 859; 2023-03-24 21:56:30,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694990420, max timestamp=1679694990437Num uniques 859; 2023-03-24 21:56:30,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:30,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:30,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679694990420, max timestamp=1679694990437Num uniques 859; 2023-03-24 21:56:30,439 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase9:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-24 21:56:30,476 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:30,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:30,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:30,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:30,479 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:30,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:30,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:30,480 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,480 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,481 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:30,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:30,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:30,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:30,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:30,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:30,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,482 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:30,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:30,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:30,483 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:30,483 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:30,485 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:30,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a93e9f5[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:30,486 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:30,486 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,488 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,488 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:30,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,488 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:30,488 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:30,488 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:30,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:30,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 509 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695050487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:30,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:30,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:30,496 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:30,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:30,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:30,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:30,496 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:30,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,497 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,498 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,499 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:30,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:30,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,502 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,502 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:30,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 511 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695050590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,602 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:30,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:30,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:30,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:30,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:30,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:30,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:30,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:30,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,608 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,608 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:30,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:30,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:30,609 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:30,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:30,610 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,610 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:30,610 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:30,610 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:30,610 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:30,614 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:30,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ef84e9[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:30,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:30,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:30,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:30,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:30,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:30,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:30,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,618 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:30,618 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:30,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:30,621 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:30,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:30,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:30,621 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:30,621 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:30,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,623 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,623 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,625 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,629 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,628 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,629 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:30,629 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:30,629 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,629 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,629 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:30,629 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:30,629 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:30,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,729 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:30,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:30,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:30,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:30,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:30,750 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:30,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:30,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:30,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,751 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,752 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,753 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:30,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:30,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:30,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:30,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:30,754 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:30,754 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:30,756 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:30,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:30,758 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f3b4623[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:30,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:30,758 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,763 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:30,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:30,763 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:30,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:30,764 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:30,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,765 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:30,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:30,767 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:30,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:30,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:30,767 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:30,767 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:30,768 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,768 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,768 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:30,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:30,769 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,770 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:30,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,772 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:30,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:30,772 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:30,773 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,773 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:30,773 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:30,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:30,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:30,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:30,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:30,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 513 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695050792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,812 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/3d3a6344b1f74bf2869e9c7df29e008e 2023-03-24 21:56:30,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=966 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9d82791a82524cee8348278877d68a8e 2023-03-24 21:56:30,837 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/96d64de0904e4457817df6db3e38243a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a 2023-03-24 21:56:30,840 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a, entries=150, sequenceid=966, filesize=25.4 K 2023-03-24 21:56:30,841 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/3d3a6344b1f74bf2869e9c7df29e008e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e 2023-03-24 21:56:30,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e, entries=150, sequenceid=966, filesize=11.9 K 2023-03-24 21:56:30,845 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/9d82791a82524cee8348278877d68a8e as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e 2023-03-24 21:56:30,848 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e, entries=150, sequenceid=966, filesize=11.9 K 2023-03-24 21:56:30,849 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 09571a8f4d95ff573f25f32562564d3a in 463ms, sequenceid=966, compaction requested=true 2023-03-24 21:56:30,849 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:30,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:30,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:30,849 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:30,849 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:30,849 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:30,850 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79432 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:30,850 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:30,850 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,850 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/da6305bbcac14ba7912447f9f49eb58e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=77.6 K 2023-03-24 21:56:30,850 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:30,850 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting da6305bbcac14ba7912447f9f49eb58e, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694989019 2023-03-24 21:56:30,850 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:30,851 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,851 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ea288690a1674ac4adb0a83bb7dfffa1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=37.0 K 2023-03-24 21:56:30,851 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 4cc74bac77ed4b8186e02ebeb110274a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1679694989125 2023-03-24 21:56:30,851 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ea288690a1674ac4adb0a83bb7dfffa1, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694989019 2023-03-24 21:56:30,851 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 96d64de0904e4457817df6db3e38243a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:30,851 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 686fa8bebc3a44798379f9c3f72e0e7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1679694989125 2023-03-24 21:56:30,852 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 3d3a6344b1f74bf2869e9c7df29e008e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:30,857 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#632 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:30,860 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#631 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:30,872 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/41dc105ad04f49969e8808f4228a80e4 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/41dc105ad04f49969e8808f4228a80e4 2023-03-24 21:56:30,873 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:30,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:30,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:30,876 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into 41dc105ad04f49969e8808f4228a80e4(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:30,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:30,876 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694990849; duration=0sec 2023-03-24 21:56:30,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:30,876 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:30,877 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37886 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:30,877 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:30,877 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,877 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4f7f1229d567447e8bf919ff5151aede, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=37.0 K 2023-03-24 21:56:30,877 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 4f7f1229d567447e8bf919ff5151aede, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=915, earliestPutTs=1679694989019 2023-03-24 21:56:30,878 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting ec0792771fae4f4c917377b88ae7543a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=952, earliestPutTs=1679694989125 2023-03-24 21:56:30,878 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 9d82791a82524cee8348278877d68a8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:30,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:30,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:30,886 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:30,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:30,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:30,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,886 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,887 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#633 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:30,889 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,889 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:30,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:30,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:30,889 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:30,889 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:30,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:30,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:30,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:30,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:30,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:30,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:30,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:30,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:30,891 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:30,891 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:30,891 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:30,891 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:30,892 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-24 21:56:30,898 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/0ebdeb0bed5f43fca625ee5e623e1cb8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0ebdeb0bed5f43fca625ee5e623e1cb8 2023-03-24 21:56:30,902 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 0ebdeb0bed5f43fca625ee5e623e1cb8(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:30,902 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:30,902 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694990849; duration=0sec 2023-03-24 21:56:30,902 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:30,917 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324e4c0bf87f3a547b382cf118295394de0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e4c0bf87f3a547b382cf118295394de0 2023-03-24 21:56:30,925 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1003, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/05ab5491248a4015bcc7f26736ad33ce 2023-03-24 21:56:31,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:31,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:31,132 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 523 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695051130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,236 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 525 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695051233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,295 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/6b40c1f271e64793bbed86ce586cc1c9 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6b40c1f271e64793bbed86ce586cc1c9 2023-03-24 21:56:31,299 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 6b40c1f271e64793bbed86ce586cc1c9(size=26.8 K), total size for store is 26.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:31,299 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:31,299 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694990849; duration=0sec 2023-03-24 21:56:31,299 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:31,337 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1003 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8b04e65aefe249a794bf7168ec36ec2d 2023-03-24 21:56:31,346 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=1003 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/66be830702a14016a05c03e694d2e30c 2023-03-24 21:56:31,349 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/05ab5491248a4015bcc7f26736ad33ce as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce 2023-03-24 21:56:31,353 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce, entries=150, sequenceid=1003, filesize=25.4 K 2023-03-24 21:56:31,353 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/8b04e65aefe249a794bf7168ec36ec2d as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d 2023-03-24 21:56:31,356 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d, entries=150, sequenceid=1003, filesize=11.9 K 2023-03-24 21:56:31,357 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/66be830702a14016a05c03e694d2e30c as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c 2023-03-24 21:56:31,359 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c, entries=150, sequenceid=1003, filesize=11.9 K 2023-03-24 21:56:31,360 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 09571a8f4d95ff573f25f32562564d3a in 469ms, sequenceid=1003, compaction requested=false 2023-03-24 21:56:31,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:31,360 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:31,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:31,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:31,360 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:31,362 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,362 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:31,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:31,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfdf094[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:31,364 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:31,364 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,365 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,365 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:31,365 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:31,365 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:31,367 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:31,367 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,367 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,367 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:31,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:31,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:31,368 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:31,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:31,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:31,368 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:31,368 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:31,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,372 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,373 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,375 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,378 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,378 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:31,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:31,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,378 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:31,378 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:31,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:31,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:31,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:31,451 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324cbd34526e7994d8ba44f0480f93ba1b6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cbd34526e7994d8ba44f0480f93ba1b6 2023-03-24 21:56:31,455 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1017, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/923898b1fc394c7dbe8cbcc1a2b02485 2023-03-24 21:56:31,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695051464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 534 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695051465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,469 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695051467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,479 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:31,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:31,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:31,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:31,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:31,481 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:31,481 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:31,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:31,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,483 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,483 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,484 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:31,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:31,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:31,485 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:31,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:31,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:31,485 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:31,485 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:31,487 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,487 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:31,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28047c9f[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:31,489 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:31,489 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,491 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,491 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:31,491 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:31,492 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:31,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,492 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:31,493 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:31,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,493 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:31,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:31,496 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:31,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:31,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:31,496 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:31,496 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:31,498 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,498 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,499 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,498 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,500 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:31,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:31,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,502 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:31,502 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:31,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:31,502 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,502 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695051568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 536 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695051568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 325 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695051570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,602 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:31,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:31,604 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:31,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:31,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:31,606 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:31,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:31,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:31,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,606 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,609 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,609 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,610 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,610 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:31,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:31,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:31,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:31,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:31,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:31,611 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:31,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:31,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:31,612 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:31,612 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:31,613 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,613 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,614 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:31,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c686dba[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:31,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:31,616 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,617 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:31,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:31,617 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:31,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,617 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:31,619 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:31,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,619 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:31,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:31,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:31,621 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:31,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:31,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:31,622 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:31,622 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:31,627 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,627 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,627 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,628 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,630 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,633 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,633 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:31,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:31,633 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,633 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,634 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,634 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:31,634 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:31,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:31,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,734 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:31,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:31,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:31,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:31,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:31,739 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:31,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:31,739 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:31,741 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,742 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,743 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,743 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,744 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:31,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:31,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,744 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:31,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:31,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:31,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:31,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,745 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:31,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:31,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:31,747 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:31,747 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:31,749 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,749 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:31,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:31,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2947384a[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:31,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:31,752 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,754 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:31,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:31,755 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:31,755 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,755 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:31,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:31,756 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,756 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:31,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:31,759 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:31,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:31,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:31,759 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:31,759 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:31,761 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,761 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,761 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,762 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,763 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:31,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,765 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,765 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:31,765 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:31,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,765 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:31,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 323 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695051770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 538 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695051770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:31,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695051774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,865 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1017 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/2e4e63d089494d4cab4c1f3546524bed 2023-03-24 21:56:31,865 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:31,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:31,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:31,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:31,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:31,869 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:31,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:31,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:31,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,870 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,872 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,872 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,873 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:31,873 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:31,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,874 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:31,874 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:31,874 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:31,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:31,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:31,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:31,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:31,874 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:31,874 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,875 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:31,875 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:31,875 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:31,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:31,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:31,875 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:31,877 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,877 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,877 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=1017 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/13134ad3534149f484584f469ee3b433 2023-03-24 21:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,877 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,877 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:31,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/923898b1fc394c7dbe8cbcc1a2b02485 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485 2023-03-24 21:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:31,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b5c14a6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:31,881 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:31,881 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,882 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:31,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:31,883 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:31,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,883 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:31,884 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:31,884 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,884 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,885 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:31,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485, entries=200, sequenceid=1017, filesize=32.3 K 2023-03-24 21:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/2e4e63d089494d4cab4c1f3546524bed as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed 2023-03-24 21:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:31,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:31,888 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:31,888 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:31,888 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:31,888 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:31,888 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:31,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed, entries=150, sequenceid=1017, filesize=11.9 K 2023-03-24 21:56:31,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/13134ad3534149f484584f469ee3b433 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433 2023-03-24 21:56:31,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,892 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:31,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,893 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:31,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433, entries=150, sequenceid=1017, filesize=11.9 K 2023-03-24 21:56:31,895 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,895 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 09571a8f4d95ff573f25f32562564d3a in 456ms, sequenceid=1017, compaction requested=true 2023-03-24 21:56:31,895 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:31,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:31,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:31,895 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:31,896 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:31,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:31,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:31,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:31,897 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86574 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:31,897 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/A is initiating minor compaction (all files) 2023-03-24 21:56:31,897 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/A in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,897 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6b40c1f271e64793bbed86ce586cc1c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=84.5 K 2023-03-24 21:56:31,897 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:31,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,897 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/B is initiating minor compaction (all files) 2023-03-24 21:56:31,897 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:31,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:31,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:31,897 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/B in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:31,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:31,897 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:31,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:31,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,897 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/41dc105ad04f49969e8808f4228a80e4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=37.1 K 2023-03-24 21:56:31,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:31,898 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:31,898 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:31,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-24 21:56:31,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:31,898 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 41dc105ad04f49969e8808f4228a80e4, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:31,898 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 6b40c1f271e64793bbed86ce586cc1c9, keycount=150, bloomtype=ROW, size=26.8 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:31,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:31,898 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:31,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:31,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:31,899 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 8b04e65aefe249a794bf7168ec36ec2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1003, earliestPutTs=1679694990475 2023-03-24 21:56:31,899 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 05ab5491248a4015bcc7f26736ad33ce, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=1003, earliestPutTs=1679694990475 2023-03-24 21:56:31,899 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] compactions.Compactor(207): Compacting 2e4e63d089494d4cab4c1f3546524bed, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1017, earliestPutTs=1679694991109 2023-03-24 21:56:31,899 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 923898b1fc394c7dbe8cbcc1a2b02485, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1017, earliestPutTs=1679694991109 2023-03-24 21:56:31,907 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#B#compaction#640 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:31,911 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#A#compaction#641 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:31,930 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/51cd517695764e2488b8e8f066e08e07 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/51cd517695764e2488b8e8f066e08e07 2023-03-24 21:56:31,934 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/A of 09571a8f4d95ff573f25f32562564d3a into 51cd517695764e2488b8e8f066e08e07(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:31,935 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:31,935 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/A, priority=13, startTime=1679694991895; duration=0sec 2023-03-24 21:56:31,935 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:31,935 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-24 21:56:31,936 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37988 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-24 21:56:31,936 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1912): 09571a8f4d95ff573f25f32562564d3a/C is initiating minor compaction (all files) 2023-03-24 21:56:31,936 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 09571a8f4d95ff573f25f32562564d3a/C in TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:31,936 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0ebdeb0bed5f43fca625ee5e623e1cb8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433] into tmpdir=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp, totalSize=37.1 K 2023-03-24 21:56:31,936 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 0ebdeb0bed5f43fca625ee5e623e1cb8, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=966, earliestPutTs=1679694990259 2023-03-24 21:56:31,936 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 66be830702a14016a05c03e694d2e30c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1003, earliestPutTs=1679694990475 2023-03-24 21:56:31,937 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] compactions.Compactor(207): Compacting 13134ad3534149f484584f469ee3b433, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1017, earliestPutTs=1679694991109 2023-03-24 21:56:31,942 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] throttle.PressureAwareThroughputController(145): 09571a8f4d95ff573f25f32562564d3a#C#compaction#642 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-24 21:56:31,957 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/8193dfe4597c4076be97ebdf52769b60 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/8193dfe4597c4076be97ebdf52769b60 2023-03-24 21:56:31,962 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/C of 09571a8f4d95ff573f25f32562564d3a into 8193dfe4597c4076be97ebdf52769b60(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:31,962 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:31,962 INFO [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/C, priority=13, startTime=1679694991896; duration=0sec 2023-03-24 21:56:31,962 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:31,998 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:31,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:32,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(933): Client=jenkins//172.31.2.10 procedure request for: flush-table-proc 2023-03-24 21:56:32,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-24 21:56:32,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-24 21:56:32,002 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-24 21:56:32,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:32,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-24 21:56:32,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-24 21:56:32,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,002 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:32,003 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:32,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:32,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,004 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-24 21:56:32,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-24 21:56:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:32,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-24 21:56:32,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-24 21:56:32,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-24 21:56:32,005 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-24 21:56:32,005 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-24 21:56:32,005 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-24 21:56:32,005 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-24 21:56:32,005 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. started... 2023-03-24 21:56:32,006 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:32,006 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,028 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032482c0145f1eb141839e829b55c4c72848 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482c0145f1eb141839e829b55c4c72848 2023-03-24 21:56:32,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:32,075 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. as already flushing 2023-03-24 21:56:32,086 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695052083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 544 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695052084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,087 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695052085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,188 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695052187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,189 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=45673] ipc.CallRunner(144): callId: 546 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695052188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=45673] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695052188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,249 DEBUG [Thread-3894] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x11947498 to 127.0.0.1:57724 2023-03-24 21:56:32,249 DEBUG [Thread-3894] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,250 DEBUG [Thread-3896] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e38d584 to 127.0.0.1:57724 2023-03-24 21:56:32,250 DEBUG [Thread-3896] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,251 DEBUG [Thread-3898] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4df4e4ba to 127.0.0.1:57724 2023-03-24 21:56:32,251 DEBUG [Thread-3898] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,252 DEBUG [Thread-3902] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0c382e3e to 127.0.0.1:57724 2023-03-24 21:56:32,252 DEBUG [Thread-3902] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,252 DEBUG [Thread-3900] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34090ce3 to 127.0.0.1:57724 2023-03-24 21:56:32,252 DEBUG [Thread-3900] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,317 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/7a82544c4a64471f93f94940790178e1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7a82544c4a64471f93f94940790178e1 2023-03-24 21:56:32,320 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 09571a8f4d95ff573f25f32562564d3a/B of 09571a8f4d95ff573f25f32562564d3a into 7a82544c4a64471f93f94940790178e1(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-24 21:56:32,320 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:32,320 INFO [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a., storeName=09571a8f4d95ff573f25f32562564d3a/B, priority=13, startTime=1679694991895; duration=0sec 2023-03-24 21:56:32,320 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:32,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 548 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56590 deadline: 1679695052390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,390 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56540 deadline: 1679695052390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,391 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-24 21:56:32,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.2.10:56554 deadline: 1679695052391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,437 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1039, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/e4b650fd42014865a42342f7ff3ff5fc 2023-03-24 21:56:32,444 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1039 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/c278f0d1e553451891ec3a5ebd9ff799 2023-03-24 21:56:32,453 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=1039 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/bc6da065df1944898f927d07bca30a5a 2023-03-24 21:56:32,456 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/e4b650fd42014865a42342f7ff3ff5fc as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e4b650fd42014865a42342f7ff3ff5fc 2023-03-24 21:56:32,458 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e4b650fd42014865a42342f7ff3ff5fc, entries=150, sequenceid=1039, filesize=25.4 K 2023-03-24 21:56:32,459 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/c278f0d1e553451891ec3a5ebd9ff799 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/c278f0d1e553451891ec3a5ebd9ff799 2023-03-24 21:56:32,461 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/c278f0d1e553451891ec3a5ebd9ff799, entries=150, sequenceid=1039, filesize=11.9 K 2023-03-24 21:56:32,462 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/bc6da065df1944898f927d07bca30a5a as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/bc6da065df1944898f927d07bca30a5a 2023-03-24 21:56:32,464 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/bc6da065df1944898f927d07bca30a5a, entries=150, sequenceid=1039, filesize=11.9 K 2023-03-24 21:56:32,465 INFO [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 09571a8f4d95ff573f25f32562564d3a in 459ms, sequenceid=1039, compaction requested=false 2023-03-24 21:56:32,465 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:32,465 DEBUG [rs(jenkins-hbase9.apache.org,45673,1679694819052)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:32,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-24 21:56:32,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-24 21:56:32,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-24 21:56:32,465 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-24 21:56:32,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,467 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:32,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,467 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-24 21:56:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:32,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase9.apache.org,45673,1679694819052' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-24 21:56:32,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403779d6[Count = 0] remaining members to acquire global barrier 2023-03-24 21:56:32,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-24 21:56:32,468 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,469 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-24 21:56:32,469 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-24 21:56:32,470 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase9.apache.org,45673,1679694819052' in zk 2023-03-24 21:56:32,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,470 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-24 21:56:32,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-24 21:56:32,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:32,472 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,472 DEBUG [member: 'jenkins-hbase9.apache.org,45673,1679694819052' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:32,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase9.apache.org,45673,1679694819052': 2023-03-24 21:56:32,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase9.apache.org,45673,1679694819052' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-24 21:56:32,474 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-24 21:56:32,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-24 21:56:32,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-24 21:56:32,474 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-24 21:56:32,474 INFO [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-24 21:56:32,478 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,478 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-24 21:56:32,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-24 21:56:32,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,478 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-24 21:56:32,479 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-24 21:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,480 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-24 21:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-24 21:56:32,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:32,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-24 21:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-24 21:56:32,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-24 21:56:32,482 DEBUG [(jenkins-hbase9.apache.org,40479,1679694817836)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-24 21:56:32,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:32,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-24 21:56:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-24 21:56:32,482 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-24 21:56:32,482 DEBUG [Thread-3893] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-24 21:56:32,482 DEBUG [Thread-3893] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-24 21:56:32,582 DEBUG [Thread-3893] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-24 21:56:32,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-24 21:56:32,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=45673] regionserver.HRegion(9158): Flush requested on 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:32,692 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-24 21:56:32,692 DEBUG [Thread-3889] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x670e081b to 127.0.0.1:57724 2023-03-24 21:56:32,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:32,693 DEBUG [Thread-3889] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:32,693 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:32,694 DEBUG [Thread-3891] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x779c2944 to 127.0.0.1:57724 2023-03-24 21:56:32,694 DEBUG [Thread-3885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x62f836a7 to 127.0.0.1:57724 2023-03-24 21:56:32,694 DEBUG [Thread-3891] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:32,694 DEBUG [Thread-3885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:33,011 DEBUG [Thread-3883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f059091 to 127.0.0.1:57724 2023-03-24 21:56:33,011 DEBUG [Thread-3883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:33,030 DEBUG [Thread-3887] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x43c7d444 to 127.0.0.1:57724 2023-03-24 21:56:33,030 DEBUG [Thread-3887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 31 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 223 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 31 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 393 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(394): wrote 226 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4319 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4253 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4242 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4338 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(398): read 4242 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-24 21:56:33,031 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:56:33,031 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x649a2bc4 to 127.0.0.1:57724 2023-03-24 21:56:33,031 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:33,032 INFO [Listener at localhost/40745] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-24 21:56:33,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$11(2418): Client=jenkins//172.31.2.10 disable TestAcidGuarantees 2023-03-24 21:56:33,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:33,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:33,035 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694993035"}]},"ts":"1679694993035"} 2023-03-24 21:56:33,036 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-24 21:56:33,037 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-24 21:56:33,038 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, UNASSIGN}] 2023-03-24 21:56:33,039 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, UNASSIGN 2023-03-24 21:56:33,040 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=CLOSING, regionLocation=jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:33,040 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694993040"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679694993040"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679694993040"}]},"ts":"1679694993040"} 2023-03-24 21:56:33,041 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052}] 2023-03-24 21:56:33,103 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230324f53a52acfd7e4a53bf31a15a286048a2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f53a52acfd7e4a53bf31a15a286048a2 2023-03-24 21:56:33,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:33,193 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(109): Close 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:33,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 09571a8f4d95ff573f25f32562564d3a, disabling compactions & flushes 2023-03-24 21:56:33,193 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:33,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:33,508 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1053, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/234f9b9509fb445ab77f477a9e1264ac 2023-03-24 21:56:33,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:33,916 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1053 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/49754546c0394d59a8c55042eb11991f 2023-03-24 21:56:34,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:34,324 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=1053 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5528ca33c5d843ab8429b179c63e1cc1 2023-03-24 21:56:34,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/234f9b9509fb445ab77f477a9e1264ac as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/234f9b9509fb445ab77f477a9e1264ac 2023-03-24 21:56:34,330 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/234f9b9509fb445ab77f477a9e1264ac, entries=150, sequenceid=1053, filesize=25.4 K 2023-03-24 21:56:34,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/49754546c0394d59a8c55042eb11991f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/49754546c0394d59a8c55042eb11991f 2023-03-24 21:56:34,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/49754546c0394d59a8c55042eb11991f, entries=150, sequenceid=1053, filesize=11.9 K 2023-03-24 21:56:34,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/5528ca33c5d843ab8429b179c63e1cc1 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5528ca33c5d843ab8429b179c63e1cc1 2023-03-24 21:56:34,337 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5528ca33c5d843ab8429b179c63e1cc1, entries=150, sequenceid=1053, filesize=11.9 K 2023-03-24 21:56:34,337 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=26.84 KB/27480 for 09571a8f4d95ff573f25f32562564d3a in 1645ms, sequenceid=1053, compaction requested=true 2023-03-24 21:56:34,337 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:34,337 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:34,337 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. because compaction request was cancelled 2023-03-24 21:56:34,337 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:34,337 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. after waiting 0 ms 2023-03-24 21:56:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:34,338 DEBUG [RS:0;jenkins-hbase9:45673-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. because compaction request was cancelled 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:34,338 DEBUG [RS:0;jenkins-hbase9:45673-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. because compaction request was cancelled 2023-03-24 21:56:34,338 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 09571a8f4d95ff573f25f32562564d3a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-24 21:56:34,338 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=A 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=B 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 09571a8f4d95ff573f25f32562564d3a, store=C 2023-03-24 21:56:34,338 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-24 21:56:34,748 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032471a691af215146288b9712c983aa97d3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471a691af215146288b9712c983aa97d3 2023-03-24 21:56:35,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:35,152 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1060, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/1c91ae778a5d4d9890e872aaf4069b63 2023-03-24 21:56:35,560 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8 2023-03-24 21:56:35,968 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1060 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/e8c824d869d24620ba2c5fcc990b5d0f 2023-03-24 21:56:35,971 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/A/1c91ae778a5d4d9890e872aaf4069b63 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1c91ae778a5d4d9890e872aaf4069b63 2023-03-24 21:56:35,974 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1c91ae778a5d4d9890e872aaf4069b63, entries=100, sequenceid=1060, filesize=18.6 K 2023-03-24 21:56:35,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8 2023-03-24 21:56:35,977 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8, entries=100, sequenceid=1060, filesize=9.5 K 2023-03-24 21:56:35,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/.tmp/C/e8c824d869d24620ba2c5fcc990b5d0f as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e8c824d869d24620ba2c5fcc990b5d0f 2023-03-24 21:56:35,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e8c824d869d24620ba2c5fcc990b5d0f, entries=100, sequenceid=1060, filesize=9.5 K 2023-03-24 21:56:35,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 09571a8f4d95ff573f25f32562564d3a in 1642ms, sequenceid=1060, compaction requested=true 2023-03-24 21:56:36,003 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7878085a61ad4b01bdcfc9debb890689, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d82ef0ca17c540269ecfb6b62c17b4eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/19ae32ed760243fc9ae2b3e797df8f15, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ee1273871c9941d79f7e3b1be4f7c6b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d0bd61976a844d1bd019073ad03b6b9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/2965a8776c13468e85a3ddcc6e079f93, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e947ec4a388041b58948f3aabb0c4f7f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c0758f4d2fd14cacb7ecee4dace2b081, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a5b693db2ffd42a7ac687e0317350fd3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/da6305bbcac14ba7912447f9f49eb58e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6b40c1f271e64793bbed86ce586cc1c9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485] to archive 2023-03-24 21:56:36,004 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:36,006 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5d6ceb02e7354be092abcec3d5e84e4c 2023-03-24 21:56:36,007 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4577419fd80a4b15b272e0ba0a5c438c 2023-03-24 21:56:36,008 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/706c85894f824bd3a76f79a7761a8561 2023-03-24 21:56:36,009 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7878085a61ad4b01bdcfc9debb890689 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7878085a61ad4b01bdcfc9debb890689 2023-03-24 21:56:36,011 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0595d22b4e47443591fd54df4eec01be 2023-03-24 21:56:36,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c4e0216dd0cf49f39fb902a65956fec4 2023-03-24 21:56:36,012 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d82ef0ca17c540269ecfb6b62c17b4eb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d82ef0ca17c540269ecfb6b62c17b4eb 2023-03-24 21:56:36,013 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/671c0094245149eeb3a2449d0148e53f 2023-03-24 21:56:36,014 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6977d6955ada41b7861e0b0cbf37be67 2023-03-24 21:56:36,015 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/129fae510c6b4ae388ab45ee1001a22b 2023-03-24 21:56:36,016 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/19ae32ed760243fc9ae2b3e797df8f15 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/19ae32ed760243fc9ae2b3e797df8f15 2023-03-24 21:56:36,017 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/0be26f3acbdd44079d5bd03387d6aeb1 2023-03-24 21:56:36,018 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/59f6d11d945c477d961e498a56ab406b 2023-03-24 21:56:36,019 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ee1273871c9941d79f7e3b1be4f7c6b0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ee1273871c9941d79f7e3b1be4f7c6b0 2023-03-24 21:56:36,020 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/7ff32124fd85489e827d46961e46fa64 2023-03-24 21:56:36,020 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/5ffbd7a7d44e4892b58e391918cdcdc3 2023-03-24 21:56:36,021 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d0bd61976a844d1bd019073ad03b6b9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d0bd61976a844d1bd019073ad03b6b9 2023-03-24 21:56:36,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/b920aba130c94e1688a088c1cb5878ea 2023-03-24 21:56:36,022 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/dbe8cbbf4a454bef8153e124f1073494 2023-03-24 21:56:36,023 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1d7ab8cf43684f74b41cea933a373ae4 2023-03-24 21:56:36,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/52bc74c27f0e451787d2940bbb88af87 2023-03-24 21:56:36,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/2965a8776c13468e85a3ddcc6e079f93 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/2965a8776c13468e85a3ddcc6e079f93 2023-03-24 21:56:36,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/eae2c7af3fbc4f4ab19f84bd1973b0cd 2023-03-24 21:56:36,026 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d1d302c3c0b840528658107447ab050e 2023-03-24 21:56:36,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e947ec4a388041b58948f3aabb0c4f7f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e947ec4a388041b58948f3aabb0c4f7f 2023-03-24 21:56:36,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/86045fcd99f34148a02b230415da0807 2023-03-24 21:56:36,028 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/8bcfd90b2619476baebe3c390fed966c 2023-03-24 21:56:36,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c0758f4d2fd14cacb7ecee4dace2b081 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/c0758f4d2fd14cacb7ecee4dace2b081 2023-03-24 21:56:36,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/cb257d56a19d4fc0879a008bce5477cc 2023-03-24 21:56:36,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ca4f7cf9c9d647d9be2fdcedf2bb8794 2023-03-24 21:56:36,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/9bd29b07618a43fc822db6413be53cd5 2023-03-24 21:56:36,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/85eedf0a20bc41e59d8b09581b84c29c 2023-03-24 21:56:36,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/ab5aa0632828451fb3e951db7b636daf 2023-03-24 21:56:36,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/797ccfd6eecc4bd085dcd6a5073c9ec8 2023-03-24 21:56:36,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a5b693db2ffd42a7ac687e0317350fd3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a5b693db2ffd42a7ac687e0317350fd3 2023-03-24 21:56:36,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/a2e8217f7b6740e1a1b75ed2086eaf9a 2023-03-24 21:56:36,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/d391fbf85ecd4d79b86fbcc11ae7e61e 2023-03-24 21:56:36,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/500809aaaf8e471c8bb3fda25a65597f 2023-03-24 21:56:36,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/da6305bbcac14ba7912447f9f49eb58e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/da6305bbcac14ba7912447f9f49eb58e 2023-03-24 21:56:36,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/4cc74bac77ed4b8186e02ebeb110274a 2023-03-24 21:56:36,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6b40c1f271e64793bbed86ce586cc1c9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/6b40c1f271e64793bbed86ce586cc1c9 2023-03-24 21:56:36,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/96d64de0904e4457817df6db3e38243a 2023-03-24 21:56:36,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/05ab5491248a4015bcc7f26736ad33ce 2023-03-24 21:56:36,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/923898b1fc394c7dbe8cbcc1a2b02485 2023-03-24 21:56:36,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a98898c77722416cbf39d509cb9b9486, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cbd466cd58374e02bc62d4708608bbb1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/e27a5d800e2248368a739319c1b31e69, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a8e6397776f54685870eb0d2301d9abe, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/31ee5e148ab94745a0c90c1bcfab3129, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b83e80911dce499084b719ec7c65399c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/d7d4c1540a63420e82b39926149420ef, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0ec859d1b69f43ec94faa38c86eb0f2c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a18eaee9db224918a3e9f4e4a0f358bf, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9b89a6425c0f41ac969796ad8b44a5fd, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ea288690a1674ac4adb0a83bb7dfffa1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/41dc105ad04f49969e8808f4228a80e4, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed] to archive 2023-03-24 21:56:36,044 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:36,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5cad76ab539849589a4ec2cf41a16fc6 2023-03-24 21:56:36,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6186129dc01243e7ad4b2b48f1a73ad9 2023-03-24 21:56:36,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a98898c77722416cbf39d509cb9b9486 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a98898c77722416cbf39d509cb9b9486 2023-03-24 21:56:36,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/95384b13e0ad44c2af4327fd2e95c810 2023-03-24 21:56:36,048 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/448e38d200954632aaf17c2c645b7e8f 2023-03-24 21:56:36,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0893d7e943214e548ccd6e27841ee654 2023-03-24 21:56:36,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cbd466cd58374e02bc62d4708608bbb1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cbd466cd58374e02bc62d4708608bbb1 2023-03-24 21:56:36,050 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f673a5fe0e0540c59a37bfe2ea62cf75 2023-03-24 21:56:36,051 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/1c6b4e2d8ee74e1186f6d01dfdf0b516 2023-03-24 21:56:36,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/146cc0ff4565495e9cdc7d1740e13086 2023-03-24 21:56:36,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/e27a5d800e2248368a739319c1b31e69 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/e27a5d800e2248368a739319c1b31e69 2023-03-24 21:56:36,054 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0de3e1f53d4b4bd09fa2c0d367318fcd 2023-03-24 21:56:36,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/797d28cf3f73496dbeed46964faf0d80 2023-03-24 21:56:36,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a8e6397776f54685870eb0d2301d9abe to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a8e6397776f54685870eb0d2301d9abe 2023-03-24 21:56:36,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/20f61cf4e39445649e17f19f5c3ac8a0 2023-03-24 21:56:36,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7790a2ff2cbc4c6ab49037820ab675c2 2023-03-24 21:56:36,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/31ee5e148ab94745a0c90c1bcfab3129 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/31ee5e148ab94745a0c90c1bcfab3129 2023-03-24 21:56:36,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8745db128ed7498a92269d320e34209a 2023-03-24 21:56:36,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/320ac414ac404a3e8eb7afbdf825e2a5 2023-03-24 21:56:36,060 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b4ebb9bc92094c11a90aecceadd012b3 2023-03-24 21:56:36,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b83e80911dce499084b719ec7c65399c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/b83e80911dce499084b719ec7c65399c 2023-03-24 21:56:36,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/6ea82f93f0be41edbbc8a134319406ef 2023-03-24 21:56:36,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/01005c9263844a859fa08c49fb014264 2023-03-24 21:56:36,063 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9a7ad5e15a524270bf2bc2aca3ca6ac4 2023-03-24 21:56:36,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/d7d4c1540a63420e82b39926149420ef to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/d7d4c1540a63420e82b39926149420ef 2023-03-24 21:56:36,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/89672d132fd040df9b80929a7bae5062 2023-03-24 21:56:36,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9146d2ef58f449ca8bf0184e0dfa506d 2023-03-24 21:56:36,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0ec859d1b69f43ec94faa38c86eb0f2c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/0ec859d1b69f43ec94faa38c86eb0f2c 2023-03-24 21:56:36,067 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/aa8918ffc77b435dbae238c36f53bd8a 2023-03-24 21:56:36,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ce2c6a95bf3e4810951be693c182093d 2023-03-24 21:56:36,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a18eaee9db224918a3e9f4e4a0f358bf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/a18eaee9db224918a3e9f4e4a0f358bf 2023-03-24 21:56:36,069 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7cba042e6dbc41c99d92182048bb2b18 2023-03-24 21:56:36,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8162645e573640cdb91ad1f66c4d5765 2023-03-24 21:56:36,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/f75f21f3cb8c41a3b7ae42852962020b 2023-03-24 21:56:36,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9b89a6425c0f41ac969796ad8b44a5fd to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/9b89a6425c0f41ac969796ad8b44a5fd 2023-03-24 21:56:36,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5d1dd3cd74d54c61b913ae178f0d1a0c 2023-03-24 21:56:36,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/5ba4e331df0c4504b7d880bf559d3fb5 2023-03-24 21:56:36,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/37085b91138d4cb7945e91d5d1c62be4 2023-03-24 21:56:36,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ea288690a1674ac4adb0a83bb7dfffa1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/ea288690a1674ac4adb0a83bb7dfffa1 2023-03-24 21:56:36,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/cd5182a7ca31437ca702685e1ed2959c 2023-03-24 21:56:36,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/686fa8bebc3a44798379f9c3f72e0e7b 2023-03-24 21:56:36,076 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/41dc105ad04f49969e8808f4228a80e4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/41dc105ad04f49969e8808f4228a80e4 2023-03-24 21:56:36,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/3d3a6344b1f74bf2869e9c7df29e008e 2023-03-24 21:56:36,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/8b04e65aefe249a794bf7168ec36ec2d 2023-03-24 21:56:36,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/2e4e63d089494d4cab4c1f3546524bed 2023-03-24 21:56:36,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f33f95cda54e43c4896c8a635414f18f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9fccf2c0bf9b4571a86cd155487fe32c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c95b69031c444f018e752eb40b4ee6b3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9960355b03534b3e9ce8e875f90fbcba, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/aea33f7ccd504a3e8541d043e316d32c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec1da27f10884f6c8ccaba286a4c93c7, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/3e62d567a20a45b68bf874d66da90a99, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4abb533ad9a14d10b20e9d30e2b84c94, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/7d61d4c179dd4212992ee46a707d7a79, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4ce7dd57ffe14109b609c961ac2755b0, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4f7f1229d567447e8bf919ff5151aede, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0ebdeb0bed5f43fca625ee5e623e1cb8, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433] to archive 2023-03-24 21:56:36,082 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-24 21:56:36,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/775dfc1100ef4142afcb382975ccd248 2023-03-24 21:56:36,084 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b70abf8fe28e475287ebc9978c72ffe3 2023-03-24 21:56:36,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f33f95cda54e43c4896c8a635414f18f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f33f95cda54e43c4896c8a635414f18f 2023-03-24 21:56:36,086 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/281113eb979d4b0296a01b08d3fc1096 2023-03-24 21:56:36,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d5daebdc8d8a47ebae7446771c17a960 2023-03-24 21:56:36,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/03ea24a9664149df832e03406a3ea3bc 2023-03-24 21:56:36,088 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9fccf2c0bf9b4571a86cd155487fe32c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9fccf2c0bf9b4571a86cd155487fe32c 2023-03-24 21:56:36,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/d14ce7ad78214054a8726353018c14a5 2023-03-24 21:56:36,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/21f6eff35eb44597b9201da41dcd8f62 2023-03-24 21:56:36,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/524686bcab4741259ee9f358697605f7 2023-03-24 21:56:36,091 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c95b69031c444f018e752eb40b4ee6b3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c95b69031c444f018e752eb40b4ee6b3 2023-03-24 21:56:36,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fd7d84e207a24251b0fb58293517cc19 2023-03-24 21:56:36,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/44b8b99dc5df4c418a94ffaa4202e53c 2023-03-24 21:56:36,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9960355b03534b3e9ce8e875f90fbcba to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9960355b03534b3e9ce8e875f90fbcba 2023-03-24 21:56:36,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb90e4ac6a21429e90b9d7b7f3bf10c5 2023-03-24 21:56:36,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4d4bfc73ad22494a859817d0571258d3 2023-03-24 21:56:36,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/aea33f7ccd504a3e8541d043e316d32c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/aea33f7ccd504a3e8541d043e316d32c 2023-03-24 21:56:36,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/295b2f5a00e644f294965d457a394e5f 2023-03-24 21:56:36,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5336c224592c4e7e894e390f192bddff 2023-03-24 21:56:36,098 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e781ba46430d4a54b4efbd19619dafcc 2023-03-24 21:56:36,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec1da27f10884f6c8ccaba286a4c93c7 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec1da27f10884f6c8ccaba286a4c93c7 2023-03-24 21:56:36,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/b62ac4c32e974fb799d36fae4a9df12c 2023-03-24 21:56:36,101 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/017eddc0739e4cf08b09b0ef73af75a1 2023-03-24 21:56:36,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9a42ddf0fba2423ba4387611d05517a5 2023-03-24 21:56:36,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/3e62d567a20a45b68bf874d66da90a99 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/3e62d567a20a45b68bf874d66da90a99 2023-03-24 21:56:36,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f4d0cfcd6ca04d32986b3d2f1148cc09 2023-03-24 21:56:36,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/a2c1aafcad9a46009ef98b3fb8a0a1eb 2023-03-24 21:56:36,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4abb533ad9a14d10b20e9d30e2b84c94 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4abb533ad9a14d10b20e9d30e2b84c94 2023-03-24 21:56:36,107 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5a97903b047c4eccbe288294fa40cdd3 2023-03-24 21:56:36,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/c6b00e8bf729442d98e8a26e63dfdbbb 2023-03-24 21:56:36,108 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/7d61d4c179dd4212992ee46a707d7a79 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/7d61d4c179dd4212992ee46a707d7a79 2023-03-24 21:56:36,109 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0da2a7b57874bae92608dab44a7514f 2023-03-24 21:56:36,110 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/af66deb7cf204b3b86dc0d5fbfbd2a44 2023-03-24 21:56:36,112 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/fb3465ea7a554d1890da13e011463e61 2023-03-24 21:56:36,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4ce7dd57ffe14109b609c961ac2755b0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4ce7dd57ffe14109b609c961ac2755b0 2023-03-24 21:56:36,113 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/f0bb884e7b6f47e2a6ef08d365128341 2023-03-24 21:56:36,114 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0241311d4d3e4c7c8019bb33aeb190e5 2023-03-24 21:56:36,115 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/32e2159d51df43ecb8fdbe90c1f11246 2023-03-24 21:56:36,116 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4f7f1229d567447e8bf919ff5151aede to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/4f7f1229d567447e8bf919ff5151aede 2023-03-24 21:56:36,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/93db49fa9f9c4626abe96ac082566886 2023-03-24 21:56:36,118 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/ec0792771fae4f4c917377b88ae7543a 2023-03-24 21:56:36,119 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0ebdeb0bed5f43fca625ee5e623e1cb8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/0ebdeb0bed5f43fca625ee5e623e1cb8 2023-03-24 21:56:36,120 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/9d82791a82524cee8348278877d68a8e 2023-03-24 21:56:36,121 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/66be830702a14016a05c03e694d2e30c 2023-03-24 21:56:36,122 DEBUG [StoreCloser-TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/13134ad3534149f484584f469ee3b433 2023-03-24 21:56:36,126 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits/1063.seqid, newMaxSeqId=1063, maxSeqId=4 2023-03-24 21:56:36,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a. 2023-03-24 21:56:36,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 09571a8f4d95ff573f25f32562564d3a: 2023-03-24 21:56:36,128 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.UnassignRegionHandler(147): Closed 09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:36,128 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=09571a8f4d95ff573f25f32562564d3a, regionState=CLOSED 2023-03-24 21:56:36,128 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679694996128"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679694996128"}]},"ts":"1679694996128"} 2023-03-24 21:56:36,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-24 21:56:36,130 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 09571a8f4d95ff573f25f32562564d3a, server=jenkins-hbase9.apache.org,45673,1679694819052 in 3.0880 sec 2023-03-24 21:56:36,131 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-24 21:56:36,131 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=09571a8f4d95ff573f25f32562564d3a, UNASSIGN in 3.0920 sec 2023-03-24 21:56:36,131 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679694996131"}]},"ts":"1679694996131"} 2023-03-24 21:56:36,132 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-24 21:56:36,133 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-24 21:56:36,134 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 3.1020 sec 2023-03-24 21:56:37,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-24 21:56:37,141 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-24 21:56:37,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.HMaster$5(2228): Client=jenkins//172.31.2.10 delete TestAcidGuarantees 2023-03-24 21:56:37,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,143 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 21:56:37,144 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,149 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:37,151 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C, FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits] 2023-03-24 21:56:37,153 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1c91ae778a5d4d9890e872aaf4069b63 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/1c91ae778a5d4d9890e872aaf4069b63 2023-03-24 21:56:37,154 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/234f9b9509fb445ab77f477a9e1264ac to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/234f9b9509fb445ab77f477a9e1264ac 2023-03-24 21:56:37,155 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/51cd517695764e2488b8e8f066e08e07 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/51cd517695764e2488b8e8f066e08e07 2023-03-24 21:56:37,156 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e4b650fd42014865a42342f7ff3ff5fc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/A/e4b650fd42014865a42342f7ff3ff5fc 2023-03-24 21:56:37,158 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/49754546c0394d59a8c55042eb11991f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/49754546c0394d59a8c55042eb11991f 2023-03-24 21:56:37,159 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/4fa3cbb18d3e4bb0ad64ae81d7e27be8 2023-03-24 21:56:37,160 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7a82544c4a64471f93f94940790178e1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/7a82544c4a64471f93f94940790178e1 2023-03-24 21:56:37,161 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/c278f0d1e553451891ec3a5ebd9ff799 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/B/c278f0d1e553451891ec3a5ebd9ff799 2023-03-24 21:56:37,163 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5528ca33c5d843ab8429b179c63e1cc1 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/5528ca33c5d843ab8429b179c63e1cc1 2023-03-24 21:56:37,164 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/8193dfe4597c4076be97ebdf52769b60 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/8193dfe4597c4076be97ebdf52769b60 2023-03-24 21:56:37,164 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/bc6da065df1944898f927d07bca30a5a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/bc6da065df1944898f927d07bca30a5a 2023-03-24 21:56:37,165 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e8c824d869d24620ba2c5fcc990b5d0f to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/C/e8c824d869d24620ba2c5fcc990b5d0f 2023-03-24 21:56:37,167 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits/1063.seqid to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a/recovered.edits/1063.seqid 2023-03-24 21:56:37,168 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/.tmp/data/default/TestAcidGuarantees/09571a8f4d95ff573f25f32562564d3a 2023-03-24 21:56:37,168 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-24 21:56:37,168 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:56:37,169 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-24 21:56:37,171 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032403c71515945e4e8bb5514898373fbff9 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032403c71515945e4e8bb5514898373fbff9 2023-03-24 21:56:37,172 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc0d229730440e5a9e6d3f9b46eea72 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240bc0d229730440e5a9e6d3f9b46eea72 2023-03-24 21:56:37,172 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240cb0c5830836441781dbf41da1e79569 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240cb0c5830836441781dbf41da1e79569 2023-03-24 21:56:37,173 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240e10ad17df1c4248ba7caded55e5faa4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303240e10ad17df1c4248ba7caded55e5faa4 2023-03-24 21:56:37,174 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324235500096eb442a0a7e21f0fc9f9f2aa to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324235500096eb442a0a7e21f0fc9f9f2aa 2023-03-24 21:56:37,175 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a8bc74cc1894ceeacce1aede92071d4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242a8bc74cc1894ceeacce1aede92071d4 2023-03-24 21:56:37,176 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c49156114b04bf6b2e571a339d3277a to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242c49156114b04bf6b2e571a339d3277a 2023-03-24 21:56:37,177 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242ca67c0cf3ea4a62a1f47be74cbec6d6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303242ca67c0cf3ea4a62a1f47be74cbec6d6 2023-03-24 21:56:37,178 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324335266e732d14b42b53ed4f9c026c46d to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324335266e732d14b42b53ed4f9c026c46d 2023-03-24 21:56:37,179 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324384d257846ed4023b2e6b3f63c2a9400 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324384d257846ed4023b2e6b3f63c2a9400 2023-03-24 21:56:37,180 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244188f0ff31eb44408663ac316f2df375 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244188f0ff31eb44408663ac316f2df375 2023-03-24 21:56:37,181 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b37da92057d405ba8e86ea4ad7d1d97 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244b37da92057d405ba8e86ea4ad7d1d97 2023-03-24 21:56:37,181 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244bce2eaa732a45abb294c8cb285bb124 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244bce2eaa732a45abb294c8cb285bb124 2023-03-24 21:56:37,182 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244c78fef0b3a94afe822f1db1edf58830 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303244c78fef0b3a94afe822f1db1edf58830 2023-03-24 21:56:37,183 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245261d6e4ee8548b883f18949d2b405ec to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303245261d6e4ee8548b883f18949d2b405ec 2023-03-24 21:56:37,185 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324579a606aa841406b8c934832b9fe8180 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324579a606aa841406b8c934832b9fe8180 2023-03-24 21:56:37,186 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246049e4f7640d4c4fa97e9cf507ea1c4e to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303246049e4f7640d4c4fa97e9cf507ea1c4e 2023-03-24 21:56:37,187 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471a691af215146288b9712c983aa97d3 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032471a691af215146288b9712c983aa97d3 2023-03-24 21:56:37,188 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324730e451ac18f4565afedeba45af34565 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324730e451ac18f4565afedeba45af34565 2023-03-24 21:56:37,188 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032474da80a1ba0f4f6c912696fb780094e8 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032474da80a1ba0f4f6c912696fb780094e8 2023-03-24 21:56:37,189 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324779e4b81646d406ea22555d3c97bbe21 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324779e4b81646d406ea22555d3c97bbe21 2023-03-24 21:56:37,190 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b93193213b043309a4b2f83357544bf to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303247b93193213b043309a4b2f83357544bf 2023-03-24 21:56:37,191 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324817160ef4b414199a00bdd85eff56859 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324817160ef4b414199a00bdd85eff56859 2023-03-24 21:56:37,192 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482c0145f1eb141839e829b55c4c72848 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032482c0145f1eb141839e829b55c4c72848 2023-03-24 21:56:37,193 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324901cfbf548dd4381b822801d6a7cb2e5 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324901cfbf548dd4381b822801d6a7cb2e5 2023-03-24 21:56:37,193 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e9ddbe6a1264f07adff5b810db9240c to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303249e9ddbe6a1264f07adff5b810db9240c 2023-03-24 21:56:37,194 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a2b30de148a94d809d342f2f3410d165 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a2b30de148a94d809d342f2f3410d165 2023-03-24 21:56:37,195 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a73f838aa6814f9f8bd32242a90609be to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324a73f838aa6814f9f8bd32242a90609be 2023-03-24 21:56:37,196 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b158ae4699ba4ab593d2b91bac111708 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324b158ae4699ba4ab593d2b91bac111708 2023-03-24 21:56:37,198 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c4cb78b04626420088dbbc061bd939cc to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324c4cb78b04626420088dbbc061bd939cc 2023-03-24 21:56:37,199 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cbd34526e7994d8ba44f0480f93ba1b6 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324cbd34526e7994d8ba44f0480f93ba1b6 2023-03-24 21:56:37,200 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324da1aeed0cdb14e6ebc702477938e1f65 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324da1aeed0cdb14e6ebc702477938e1f65 2023-03-24 21:56:37,201 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e08032116c974601844772ffd57366c4 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e08032116c974601844772ffd57366c4 2023-03-24 21:56:37,201 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e4c0bf87f3a547b382cf118295394de0 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324e4c0bf87f3a547b382cf118295394de0 2023-03-24 21:56:37,202 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f53a52acfd7e4a53bf31a15a286048a2 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324f53a52acfd7e4a53bf31a15a286048a2 2023-03-24 21:56:37,203 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fd8cf31ab18147c1a2178619a51b0966 to hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230324fd8cf31ab18147c1a2178619a51b0966 2023-03-24 21:56:37,204 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-24 21:56:37,205 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,207 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-24 21:56:37,208 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-24 21:56:37,209 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,209 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-24 21:56:37,209 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679694997209"}]},"ts":"9223372036854775807"} 2023-03-24 21:56:37,211 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-24 21:56:37,211 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 09571a8f4d95ff573f25f32562564d3a, NAME => 'TestAcidGuarantees,,1679694969162.09571a8f4d95ff573f25f32562564d3a.', STARTKEY => '', ENDKEY => ''}] 2023-03-24 21:56:37,211 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-24 21:56:37,211 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679694997211"}]},"ts":"9223372036854775807"} 2023-03-24 21:56:37,212 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-24 21:56:37,214 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-24 21:56:37,214 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-03-24 21:56:37,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=40479] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-24 21:56:37,245 INFO [Listener at localhost/40745] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-24 21:56:37,259 INFO [Listener at localhost/40745] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 225) - Thread LEAK? -, OpenFileDescriptor=608 (was 586) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=446 (was 446), ProcessCount=169 (was 169), AvailableMemoryMB=8371 (was 8285) - AvailableMemoryMB LEAK? - 2023-03-24 21:56:37,260 INFO [Listener at localhost/40745] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-24 21:56:37,260 INFO [Listener at localhost/40745] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-24 21:56:37,260 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0904a3c5 to 127.0.0.1:57724 2023-03-24 21:56:37,261 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:37,261 DEBUG [Listener at localhost/40745] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-24 21:56:37,261 DEBUG [Listener at localhost/40745] util.JVMClusterUtil(257): Found active master hash=63376126, stopped=false 2023-03-24 21:56:37,261 INFO [Listener at localhost/40745] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:56:37,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 21:56:37,263 INFO [Listener at localhost/40745] procedure2.ProcedureExecutor(629): Stopping 2023-03-24 21:56:37,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:56:37,263 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-24 21:56:37,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 21:56:37,264 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-24 21:56:37,263 DEBUG [Listener at localhost/40745] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x78fa30d3 to 127.0.0.1:57724 2023-03-24 21:56:37,264 DEBUG [Listener at localhost/40745] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:37,264 INFO [Listener at localhost/40745] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,45673,1679694819052' ***** 2023-03-24 21:56:37,264 INFO [Listener at localhost/40745] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-24 21:56:37,264 INFO [RS:0;jenkins-hbase9:45673] regionserver.HeapMemoryManager(220): Stopping 2023-03-24 21:56:37,264 INFO [RS:0;jenkins-hbase9:45673] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-24 21:56:37,264 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-24 21:56:37,264 INFO [RS:0;jenkins-hbase9:45673] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-24 21:56:37,265 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(3304): Received CLOSE for 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:56:37,266 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:37,266 DEBUG [RS:0;jenkins-hbase9:45673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1217d297 to 127.0.0.1:57724 2023-03-24 21:56:37,266 DEBUG [RS:0;jenkins-hbase9:45673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:37,266 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 40b7a2bac1d0cd056af86198ad6d9f0a, disabling compactions & flushes 2023-03-24 21:56:37,267 INFO [RS:0;jenkins-hbase9:45673] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-24 21:56:37,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:56:37,267 INFO [RS:0;jenkins-hbase9:45673] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-24 21:56:37,267 INFO [RS:0;jenkins-hbase9:45673] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-24 21:56:37,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:56:37,267 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-24 21:56:37,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. after waiting 0 ms 2023-03-24 21:56:37,267 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:56:37,267 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 40b7a2bac1d0cd056af86198ad6d9f0a 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-24 21:56:37,267 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-24 21:56:37,267 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1479): Online Regions={40b7a2bac1d0cd056af86198ad6d9f0a=hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a., 1588230740=hbase:meta,,1.1588230740} 2023-03-24 21:56:37,267 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-24 21:56:37,268 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-24 21:56:37,268 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-24 21:56:37,268 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-24 21:56:37,268 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-24 21:56:37,268 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-24 21:56:37,269 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740, 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:56:37,355 INFO [regionserver/jenkins-hbase9:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-24 21:56:37,469 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740, 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:56:37,669 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740, 40b7a2bac1d0cd056af86198ad6d9f0a 2023-03-24 21:56:37,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/.tmp/info/731350bc75704618aef5838d07f11a11 2023-03-24 21:56:37,681 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/info/fa54a0d107904e2c9a1f449d632b9a66 2023-03-24 21:56:37,685 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fa54a0d107904e2c9a1f449d632b9a66 2023-03-24 21:56:37,686 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/.tmp/info/731350bc75704618aef5838d07f11a11 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/info/731350bc75704618aef5838d07f11a11 2023-03-24 21:56:37,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/info/731350bc75704618aef5838d07f11a11, entries=2, sequenceid=6, filesize=4.8 K 2023-03-24 21:56:37,690 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 40b7a2bac1d0cd056af86198ad6d9f0a in 423ms, sequenceid=6, compaction requested=false 2023-03-24 21:56:37,696 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/namespace/40b7a2bac1d0cd056af86198ad6d9f0a/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-24 21:56:37,697 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:56:37,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 40b7a2bac1d0cd056af86198ad6d9f0a: 2023-03-24 21:56:37,697 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679694822065.40b7a2bac1d0cd056af86198ad6d9f0a. 2023-03-24 21:56:37,870 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 21:56:38,031 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-24 21:56:38,031 INFO [regionserver/jenkins-hbase9:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-24 21:56:38,070 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 21:56:38,095 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/rep_barrier/5358277318f344629a668c617807b5d2 2023-03-24 21:56:38,098 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5358277318f344629a668c617807b5d2 2023-03-24 21:56:38,270 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-03-24 21:56:38,270 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-03-24 21:56:38,270 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 21:56:38,471 DEBUG [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-24 21:56:38,504 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/table/cfc1020c5fd14257bd6bc812fc8afa93 2023-03-24 21:56:38,506 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfc1020c5fd14257bd6bc812fc8afa93 2023-03-24 21:56:38,507 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/info/fa54a0d107904e2c9a1f449d632b9a66 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/info/fa54a0d107904e2c9a1f449d632b9a66 2023-03-24 21:56:38,510 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for fa54a0d107904e2c9a1f449d632b9a66 2023-03-24 21:56:38,510 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/info/fa54a0d107904e2c9a1f449d632b9a66, entries=22, sequenceid=93, filesize=7.1 K 2023-03-24 21:56:38,512 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/rep_barrier/5358277318f344629a668c617807b5d2 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/rep_barrier/5358277318f344629a668c617807b5d2 2023-03-24 21:56:38,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 5358277318f344629a668c617807b5d2 2023-03-24 21:56:38,514 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/rep_barrier/5358277318f344629a668c617807b5d2, entries=6, sequenceid=93, filesize=5.5 K 2023-03-24 21:56:38,515 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/.tmp/table/cfc1020c5fd14257bd6bc812fc8afa93 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/table/cfc1020c5fd14257bd6bc812fc8afa93 2023-03-24 21:56:38,517 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for cfc1020c5fd14257bd6bc812fc8afa93 2023-03-24 21:56:38,518 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/table/cfc1020c5fd14257bd6bc812fc8afa93, entries=9, sequenceid=93, filesize=5.5 K 2023-03-24 21:56:38,518 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1250ms, sequenceid=93, compaction requested=false 2023-03-24 21:56:38,527 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-24 21:56:38,527 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-24 21:56:38,528 INFO [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-24 21:56:38,528 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-24 21:56:38,529 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase9:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-24 21:56:38,671 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,45673,1679694819052; all regions closed. 2023-03-24 21:56:38,679 DEBUG [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/oldWALs 2023-03-24 21:56:38,679 INFO [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C45673%2C1679694819052.meta:.meta(num 1679694821791) 2023-03-24 21:56:38,683 DEBUG [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/oldWALs 2023-03-24 21:56:38,683 INFO [RS:0;jenkins-hbase9:45673] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase9.apache.org%2C45673%2C1679694819052:(num 1679694821178) 2023-03-24 21:56:38,683 DEBUG [RS:0;jenkins-hbase9:45673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:38,683 INFO [RS:0;jenkins-hbase9:45673] regionserver.LeaseManager(133): Closed leases 2023-03-24 21:56:38,683 INFO [RS:0;jenkins-hbase9:45673] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase9:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-03-24 21:56:38,684 INFO [regionserver/jenkins-hbase9:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 21:56:38,685 INFO [RS:0;jenkins-hbase9:45673] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:45673 2023-03-24 21:56:38,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase9.apache.org,45673,1679694819052 2023-03-24 21:56:38,691 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 21:56:38,691 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1b4c1999 rejected from java.util.concurrent.ThreadPoolExecutor@1f739e28[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,692 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-24 21:56:38,692 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@109a0c81 rejected from java.util.concurrent.ThreadPoolExecutor@1f739e28[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,692 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase9.apache.org,45673,1679694819052] 2023-03-24 21:56:38,693 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase9.apache.org,45673,1679694819052; numProcessing=1 2023-03-24 21:56:38,694 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase9.apache.org,45673,1679694819052 already deleted, retry=false 2023-03-24 21:56:38,694 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase9.apache.org,45673,1679694819052 expired; onlineServers=0 2023-03-24 21:56:38,694 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase9.apache.org,40479,1679694817836' ***** 2023-03-24 21:56:38,694 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-24 21:56:38,694 DEBUG [M:0;jenkins-hbase9:40479] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@42fa00c5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase9.apache.org/172.31.2.10:0 2023-03-24 21:56:38,695 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegionServer(1145): stopping server jenkins-hbase9.apache.org,40479,1679694817836 2023-03-24 21:56:38,695 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegionServer(1171): stopping server jenkins-hbase9.apache.org,40479,1679694817836; all regions closed. 2023-03-24 21:56:38,695 DEBUG [M:0;jenkins-hbase9:40479] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-24 21:56:38,695 DEBUG [M:0;jenkins-hbase9:40479] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-24 21:56:38,695 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-24 21:56:38,695 DEBUG [M:0;jenkins-hbase9:40479] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-24 21:56:38,695 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1679694820923] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.large.0-1679694820923,5,FailOnTimeoutGroup] 2023-03-24 21:56:38,695 DEBUG [master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1679694820924] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase9:0:becomeActiveMaster-HFileCleaner.small.0-1679694820924,5,FailOnTimeoutGroup] 2023-03-24 21:56:38,695 INFO [M:0;jenkins-hbase9:40479] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-24 21:56:38,695 INFO [M:0;jenkins-hbase9:40479] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-24 21:56:38,695 INFO [M:0;jenkins-hbase9:40479] hbase.ChoreService(369): Chore service for: master/jenkins-hbase9:0 had [] on shutdown 2023-03-24 21:56:38,696 DEBUG [M:0;jenkins-hbase9:40479] master.HMaster(1512): Stopping service threads 2023-03-24 21:56:38,696 INFO [M:0;jenkins-hbase9:40479] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-24 21:56:38,696 ERROR [M:0;jenkins-hbase9:40479] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-11,5,PEWorkerGroup] Thread[HFileArchiver-12,5,PEWorkerGroup] Thread[IPC Client (1027352456) connection to localhost/127.0.0.1:45841 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-24 21:56:38,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-24 21:56:38,696 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-24 21:56:38,696 INFO [M:0;jenkins-hbase9:40479] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-24 21:56:38,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-24 21:56:38,697 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-24 21:56:38,697 DEBUG [M:0;jenkins-hbase9:40479] zookeeper.ZKUtil(398): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-24 21:56:38,697 WARN [M:0;jenkins-hbase9:40479] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-24 21:56:38,697 INFO [M:0;jenkins-hbase9:40479] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-24 21:56:38,697 INFO [M:0;jenkins-hbase9:40479] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-24 21:56:38,698 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-24 21:56:38,698 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:56:38,698 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:56:38,698 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-24 21:56:38,698 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:56:38,698 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.71 KB heapSize=393.15 KB 2023-03-24 21:56:38,710 INFO [M:0;jenkins-hbase9:40479] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.71 KB at sequenceid=574 (bloomFilter=true), to=hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/533656bcfc8943eab8eebf925c418477 2023-03-24 21:56:38,714 INFO [M:0;jenkins-hbase9:40479] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 533656bcfc8943eab8eebf925c418477 2023-03-24 21:56:38,714 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/533656bcfc8943eab8eebf925c418477 as hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/533656bcfc8943eab8eebf925c418477 2023-03-24 21:56:38,717 INFO [M:0;jenkins-hbase9:40479] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 533656bcfc8943eab8eebf925c418477 2023-03-24 21:56:38,717 INFO [M:0;jenkins-hbase9:40479] regionserver.HStore(1080): Added hdfs://localhost:45841/user/jenkins/test-data/d03395ec-4a56-afec-c176-669c0957d420/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/533656bcfc8943eab8eebf925c418477, entries=65, sequenceid=574, filesize=24.7 K 2023-03-24 21:56:38,718 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegion(2948): Finished flush of dataSize ~344.71 KB/352983, heapSize ~393.13 KB/402568, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 20ms, sequenceid=574, compaction requested=false 2023-03-24 21:56:38,720 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-24 21:56:38,720 DEBUG [M:0;jenkins-hbase9:40479] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-24 21:56:38,722 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-24 21:56:38,722 INFO [M:0;jenkins-hbase9:40479] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-24 21:56:38,723 INFO [M:0;jenkins-hbase9:40479] ipc.NettyRpcServer(158): Stopping server on /172.31.2.10:40479 2023-03-24 21:56:38,725 DEBUG [M:0;jenkins-hbase9:40479] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase9.apache.org,40479,1679694817836 already deleted, retry=false 2023-03-24 21:56:38,793 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 21:56:38,793 INFO [RS:0;jenkins-hbase9:45673] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,45673,1679694819052; zookeeper connection closed. 2023-03-24 21:56:38,793 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@68600e40 rejected from java.util.concurrent.ThreadPoolExecutor@1f739e28[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,794 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): regionserver:45673-0x1011885c6c20001, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 21:56:38,794 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@3626fe73] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@3626fe73 2023-03-24 21:56:38,794 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ae00123 rejected from java.util.concurrent.ThreadPoolExecutor@1f739e28[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2906] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,794 INFO [Listener at localhost/40745] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-24 21:56:38,893 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 21:56:38,893 INFO [M:0;jenkins-hbase9:40479] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase9.apache.org,40479,1679694817836; zookeeper connection closed. 2023-03-24 21:56:38,893 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@46e18f9b rejected from java.util.concurrent.ThreadPoolExecutor@262a300[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4854] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,893 DEBUG [Listener at localhost/40745-EventThread] zookeeper.ZKWatcher(600): master:40479-0x1011885c6c20000, quorum=127.0.0.1:57724, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-24 21:56:38,893 ERROR [Listener at localhost/40745-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@445c03df rejected from java.util.concurrent.ThreadPoolExecutor@262a300[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4854] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-24 21:56:38,895 WARN [Listener at localhost/40745] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-24 21:56:38,898 INFO [Listener at localhost/40745] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 21:56:39,007 WARN [BP-1653792093-172.31.2.10-1679694814530 heartbeating to localhost/127.0.0.1:45841] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-03-24 21:56:39,007 WARN [BP-1653792093-172.31.2.10-1679694814530 heartbeating to localhost/127.0.0.1:45841] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1653792093-172.31.2.10-1679694814530 (Datanode Uuid 6db8e130-b292-4c29-b205-f31c95509b69) service to localhost/127.0.0.1:45841 2023-03-24 21:56:39,012 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de/dfs/data/data1/current/BP-1653792093-172.31.2.10-1679694814530] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 21:56:39,013 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/b5cedaac-14d9-68e7-ef5c-edd20bb83430/cluster_79d5d920-1731-b86c-549d-7cf975b552de/dfs/data/data2/current/BP-1653792093-172.31.2.10-1679694814530] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-24 21:56:39,058 INFO [Listener at localhost/40745] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-24 21:56:39,229 INFO [Listener at localhost/40745] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-24 21:56:39,409 INFO [Listener at localhost/40745] hbase.HBaseTestingUtility(1293): Minicluster is down